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Abstract 

A  series  of  scenarios  is  investigated  whereby  an  optical  flow  -  balancing  guidance 
law  is  used  to  avoid  obstacles  by  steering  an  air  vehicle  around  or  between  fixed 
objects.  These  obstacles  are  registered  as  specific  points  that  are  representative  of 
features  in  a  scene.  The  obstacles  appear  in  the  field  of  view  of  a  single  forward  looking 
camera.  Ideally,  the  guidance  law  should  allow  the  vehicle  to  pass  safely  between  the 
objects  on  a  trajectory  that  passes  through  the  orthogonal  bisector  of  the  segment 
connecting  the  two  closest  obstacles.  First,  a  2-D  analysis  is  presented  where  the  rate 
of  the  line  of  sight  from  the  vehicle  to  each  of  the  obstacles  to  be  avoided  is  measured. 
The  analysis  proceeds  by  using  no  field  of  view  (FOV)  limitations,  then  applying 
FOV  restrictions,  and  then  adding  features  or  obstacles  in  the  scene.  These  analyses 
show  that  using  a  guidance  law  that  equalizes  the  line  of  sight  rates  with  no  FOV 
limitations  results  in  the  vehicle  being  steered  into  one  of  the  objects  for  all  initial 
conditions.  An  exception  is  when  the  vehicle  is  initially  on,  and  aligned  with,  the 
orthogonal  bisector.  Even  in  this  most  favorable  instance,  the  trajectory  is  not  stable 
and  a  collision  with  an  obstacle  might  ensue.  The  research  next  develops  an  obstacle 
avoidance  strategy  based  on  equilibrating  the  optic  flow  generated  by  the  obstacles 
and  presents  an  analysis  that  leads  to  a  different  conclusion  in  which  balancing  the 
optic  flows  does  avoid  the  obstacles.  The  special  symmetric  case  is  analyzed  where 
the  vehicle’s  trajectory  is  on  the  orthogonal  bisector  of  the  segment  connecting  the 
two  obstacles,  the  ideal  obstacle  avoidance  trajectory  which  balances  optic  flow  or 
the  rate  of  the  lines  of  sight  to  the  obstacles.  An  analysis  is  presented  that  shows 
that  this  special  trajectory  is  unstable  for  the  linearized  case,  but  the  obstacles  are 
avoided.  The  paper  then  describes  a  set  of  guidance  methods  that  with  real  FOV 
limitations  create  a  favorable  result.  Finally,  the  looming  of  an  object  in  the  camera’s 
FOV  can  be  measured  and  used  for  synthesizing  a  collision  avoidance  guidance  law. 


IV 


For  the  simple  2-D  case,  looming  is  quantified  as  an  increase  in  LOS  between  two 
features  on  a  wall  in  front  of  the  air  vehicle.  The  2-D  guidance  law  for  equalizing 
the  optic  flow  and  looming  detection  is  then  extended  into  the  3-D  case.  Then  a  set 
of  3-D  scenarios  are  further  explored  using  a  decoupled  two  channel  approach,  the 
horizontal  and  vertical  channels.  In  addition,  the  3-D  scenarios  compare  two  image 
segmentation  techniques  that  are  used  to  find  optic  flow  vectors  that  could  be  used  to 
find  obstacles  in  an  image.  Finally,  the  results  from  multiple  scenarios  will  prove  that 
3-D  obstacle  avoidance  is  possible  with  proper  image  processing  tools  and  applying 
the  aforementioned  LOS  rate  equalization  and  looming  detection  guidance  laws. 
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Collision  Avoidance 


for  UAVs 

Using  Optic  Flow  Measurement 
With  Line  of  Sight  Rate  Equalization 
and  Looming 

I.  Introduction 

1 . 1  Motivation 

Collision  avoidance  is  a  basic  necessity.  Anything  in  motion,  for  example  a  school 
of  fish  or  a  flock  of  birds,  must  practice  collision  avoidance.  “Collision  avoidance  is  the 
collision- free  movement  of  two  or  more  objects  [9j.”  There  are  two  primary  techniques 
used  to  accomplish  this.  The  first  is  continuous  path  planning  for  the  avoidance  of 
known  obstacles  and  the  second  is  on  the  fly  collision  detection  [9].  Path  planning 
involves  a  priori  knowledge  of  all  obstacles,  i.e.,  buildings,  bridges,  and  towers.  It  also 
necessitates  knowing  the  vehicles  velocity.  Then  a  path  is  planned  by  mathematical 
and  analytical  calculations  to  avoid  the  known  obstacles.  The  collision  avoidance 
module  uses  vision  and  algorithms  to  check  if  static  objects  are  obstacles  where  a 
collision  is  possible  and  then  takes  avoidance  action. 

In  UAV  operations,  collision  avoidance  is  a  necessity.  Many  Micro  Unmanned 
Air  Vehicle  (MAV) /Unmanned  Air  Vehicles  (UAV)  operate  within  urban  canyons  and 
in  this  unknown  environment  many  obstacles  like  buildings,  trees,  power  lines,  and 
possibly  other  man-made  obstacles  exist.  Their  ability  to  navigate  safely  and  au¬ 
tonomously  in  this  environment  without  collision  is  key  to  mission  success.  There  is 
also  the  chance  that  a  UAV’s  planned  flight  path  take  it  into  higher  elevations  and 
into  commercial  airspace.  In  this  airspace,  the  Federal  Aviation  Administration(FAA) 
has  stringent  guidelines  on  collision  avoidance  that  all  aircraft  must  be  able  to  comply 
with  [7j.  This  makes  collision  avoidance  a  fundamental  part  of  a  UAV’s  flight.  One  of 
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the  keys  to  autonomous  UAV  flight  comes  from  the  ability  to  perform  collision  avoid¬ 
ance  in  real-time.  All  aircraft  must  accomplish  this  in  some  way.  For  manned  aircraft, 
this  is  done  visually  or  by  transponder  in  a  cooperative  traffic  environment  [2H]  •  Addi¬ 
tional  solutions  to  collision  avoidance  include  radar  and  laser  ranging.  For  MAVs  this 
is  not  feasible  due  to  size  and  weight  constraints.  The  hardware  necessary  for  these 
methods  is  far  too  heavy  for  most  MAVs,  hence  the  camera  which  can  be  lightweight 
and  is  already  flying  on  many  MAVs  is  the  sensor  of  opportunity  [3].  The  camera 
being  the  sensor  of  choice  means  that  exploitation  of  video  is  the  obvious  answer  to 
collision  avoidance  in  these  types  of  aircraft.  The  following  sections  describe  some 
actual  applications  that  this  research  could  directly  impact. 


1.1.1  Military  Applications.  The  applications  for  UAVs  have  expanded  in 
recent  years  [2j|.  Some  have  been  fitted  with  various  sensors  including  cameras.  Us¬ 
ing  the  images  from  a  camera  mounted  on  a  UAV  has  had  a  tremendous  impact  on 
how  the  military  does  business.  There  are  several  UAV  applications  that  the  military 
uses,  for  example:  Intelligence  Surveillance  and  Reconnaissance  (ISR)  missions  like 
convoy  surveillance,  target  localization,  target  prosecution,  communications  transmis¬ 
sions  and  border  and  port  surveillance  0  0.  These  applications  are  appealing  to  the 
military  as  these  missions  can  be  executed  without  risking  human  lives  0.  Saving 
lives,  while  still  meeting  the  missions  of  the  military,  is  a  definite  objective.  In  recent 
years,  the  UAV  mission  has  been  expanded  to  include  weapons  delivery,  such  as  the 
hellfire  missile  [2].  There  is  also  an  appeal  because  UAVs  have  small  radar  cross  sec¬ 
tions  and  are  difficult  to  detect  |2j .  Accomplishing  ISR  tasks  without  detection  gives 
the  military  a  distinct  advantage. 


UAVs  are  already  an  important  part  of  the  U.S.  Air  Force’s  inventory:  Global 
Hawk,  Predator,  Hunter,  Black  Widow,  and  the  ’BATCAM’  MAV.  While  Global 
Hawk  is  one  of  the  largest  UAV  at  25,000  pounds  there  are  also  smaller  ones  like  the 
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Black  Widow  at  a  half  pound  |2] .  These  type  of  unmanned  air  vehicles  are  the  future 
of  modern  warfare. 


1.1.2  Commercial  Applications.  The  UAV  is  not  just  limited  to  military  ap¬ 
plications.  Civilian  applications  are  also  possible.  Many  civilian  organizations  could 
use,  or  are  using,  UAV/MAVs  for  various  tasks.  Some  of  these  tasks  include:  mon¬ 
itoring  critical  infrastructure,  real-time  disaster  observation,  wilderness  search  and 
rescue,  and  in-storm  weather  measurements  [7j;  once  again  saving  lives.  Flying  air¬ 
craft  into  dangerous  environments  like  over  forest  fires  or  large  chemical  spills  could 
then  be  avoided  by  piloted  aircraft.  It  can  even  be  envisioned  that  some  typical  he¬ 
licopter  or  blimp  operations  could  give  way  to  the  UAV.  Police  could  use  UAVs  for 
persistent  suspect  surveillance,  tracking  high  speed  car  chases,  or  even  monitoring 
protests,  demonstrations,  and  riots  [7j.  These  could  be  accomplished  without  a  pilot 
and  large  and  expensive  aircraft.  The  UAV  is  much  smaller,  costs  less,  and  requires 
less  maintenance  than  an  aircraft  or  helicopter.  This  makes  the  employment  of  MAVs 
appealing  to  the  civilian  sector  as  well  as  the  military. 


1.2  Problem  Statement:  Collision  Avoidance 

Unmanned  Aerial  Vehicles  (UAVs)  and  Micro-UAV(MAV)  have  become  increas¬ 
ingly  more  useful.  The  following  research  develops  a  theory  to  support  the  UAV/- 
MAV’s  capability  to  navigate  autonomously  and  avoid  obstacles.  The  ability  for  a 
UAV  to  navigate  through  its  environment  is  key  for  survivability  and  mission  success. 
This  can  only  be  done  by  designing  a  collision  avoidance  guidance  and  control  mod¬ 
ule.  Ensuring  UAVs  are  autonomous  implies  that  collision  avoidance  must  be  done 
autonomously  as  well  |3J .  This  research  develops  an  algorithm  that  detects  features 
in  a  camera’s  Field  of  View(FOV)  from  a  forward  mounted  camera  on  a  UAV.  This 
means  that  only  objects  directly  in  the  vehicle’s  flight  path  are  detected.  The  algo- 
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rithm  uses  the  measurement  of  optic  flow  for  line  of  sight  rate  equalization  and  also 
looming  detection  to  determine  proper  course  corrections  so  that  the  vehicle  does  not 
collide  with  any  obstacles.  This  vision-based  collision  avoidance  algorithm  is  key  for 
the  UAV’s  survivability  and  its  acceptance  by  operators.  Survivability  and  autonomy 
of  these  ISR  assets  is  necessary  to  operate  whenever  they  are  needed  [3] .  The  follow¬ 
ing  describes  the  approach  to  the  problem  and  techniques  used  in  the  solution.  This 
approach  entails  the  method  of  line  of  sight  rate  equalization  and  looming  detection 
in  order  to  achieve  collision  free  navigation. 


1.3  Approach 

The  research  provided  is  followed  by  a  simulation  study.  The  first  part  im¬ 
plements  collision  avoidance  using  line  of  sight  (LOS)  rate  equalization  and  looming 
detection  using  the  measurement  of  optic  flow  in  a  two  dimensional  (2-D)  scenario. 
The  simulation  supplies  points  as  features  tracked  over  time,  thus  generating  optic 
flow.  The  features  are  found  in  different  configurations  and  the  guidance  algorithm 
is  applied  to  provide  collision  avoidance.  The  2-D  simulation  confirms  that  the  con¬ 
trol  law  employed  works  as  expected.  Next,  a  three  dimensional  (3-D)  simulation 
is  implemented  with  the  devised  control  law  based  on  LOS  rate  equalization  and 
looming  detection.  The  3-D  collision  avoidance  is  handled  as  two  decoupled  channel 
guidance  problems,  the  horizontal  and  vertical  channels.  In  each  channel,  a  version 
of  the  2-D  control  algorithm  is  applied  using  image  processing  techniques.  The  3-D 
environment  has  randomly  placed  buildings  and  obstacles  that  an  air  vehicle  has  to 
navigate  through.  In  the  following  subsections,  the  LOS  rate  equalization  and  looming 
detection  algorithms  are  described  in  more  detail. 

1.3.1  Line  of  Sight  Rate  Equalization.  The  measurement  of  optic  flow  has 
been  known  for  a  long  time  na-  Optic  flow  is  defined  as  “the  distribution  of  apparent 
velocities  of  movement  of  brightness  patterns  in  an  image  H3”  It  can  be  caused  by 
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an  object’s  motion,  by  the  observer’s  motion,  or  both  [15.J.  Optic  flow  is  visualized  as 
a  two  dimensional  vector  field  of  velocities,  the  optic  flow  field  [22].  This  optic  flow 
field  generally  arises  from  translational  or  angular  movements  [22]  •  These  flow  fields 
will  give  information  on  the  magnitude  and  direction  of  motion.  In  this  work,  optic 
flow  velocity  fields  are  used  for  obstacle  detection  in  a  UAV’s  FOV  and  the  ensuing 
obstacle  avoidance.  These  right  and  left  hand  obstacles’  flow  vectors  are  then  equal¬ 
ized  by  a  feedback  control  which  guides  the  vehicle  toward  the  orthogonal  bisector  of 
the  line  which  connects  clusters  of  obstacles  to  the  right  and  to  the  left  of  the  MAV. 
The  guidance  is  based  on  LOS  rate  measurement  to  each  object.  The  line  of  sight 
rate  to  each  object  is  calculated  and  the  heading  of  the  UAV  is  adjusted  accordingly. 
This  will  ensure  that  the  vehicle  flies  through  the  clusters  of  obstacles. 


1.3.2  Looming.  Looming  is  known  as  the  apparent  growth  in  size  of  an 
object  which  is  viewed  over  time.  The  visual  looming  effect  is  characterized  by  “the 
expansion  of  the  projection  of  an  object  in  the  retina  eg.”  The  visual  cue  of  an 
object  growing  larger  is  a  stimulus  in  animals,  including  humans,  that  usually  results 
in  a  change  in  direction.  This  is  one  of  the  primary  cues  in  humans  and  is  a  reflex  for 
obstacle  avoidance  [18j.  In  this  work,  looming  will  be  detected  using  optic  flow  and 
by  monitoring  the  optic  flow  for  several  frames.  With  optic  flow,  the  looming  effect 
is  defined  by  flow  vectors  diverging  from  a  single  point  in  an  image  called  the  focus 
of  expansion  (FOE)  [22].  These  flow  vectors  are  segmented  and  checked  for  growth 
or  continued  divergence.  If  this  divergence  is  maintained  for  successive  frames  and 
the  FOE  is  centered  in  the  image,  it  is  determined  that  a  collision  is  imminent.  From 
this,  a  decision  is  made  to  avoid  the  obstacle.  This  is  the  highest  priority  control  law, 
given  that  an  object  must  be  in  the  vehicle’s  direct  path.  This  technique  is  shown  to 
be  effective  for  obstacle  avoidance. 


5 


1.4  Summary 

The  research  will  demonstrate  that  a  viable  solution  for  collision  avoidance  can 
be  designed  using  LOS  rate  equalization  and  looming  detection  using  the  measurement 
of  optic  flow.  Providing  a  solution  for  collision  detection  and  avoidance  is  critical  to 
the  autonomous  capability  of  the  UAV.  As  previously  mentioned,  collision  avoidance 
can  be  accomplished  in  two  ways.  The  first  is  accurate  path  planning  which  uses  a 
priori  information  and  on-board  sensors  like  GPS  to  determine  a  path  to  navigate 
through  an  environment.  The  second  is  detection  of  unknown  objects  in  real-time, 
on-the-fly,  and  control  the  UAV  to  avoid  these  objects.  The  second  part  is  addressed 
in  this  research.  This  research  will  show  that  this  could  be  achieved  with  low  cost 
video  cameras  and  image  processing.  The  theories  developed  can  be  adapted  easily  to 
actual  image  sequences  with  optic  flow  algorithms  already  in  use  for  real-time  collision 
avoidance  in  UAV’s. 

In  Chapter  an  the  theories  relating  to  this  thesis  are  discussed.  The  chapter 
begins  with  the  discussion  of  Horn  Schunck  optic  flow  and  common  methods  for 
developing  flow  fields.  Next  it  provides  a  biomimetic  study  on  insects  that  gives 
insight  into  simple  vision  systems  occurring  in  nature.  This  is  done  for  the  purpose 
of  exploitation  in  computer  vision  systems.  Next,  the  concept  of  looming  is  described 
and  some  research  is  cited  that  uses  looming  in  robotic  systems.  The  research  of 
applying  an  optic  flow  collision  avoidance  technique  is  identified  and  is  analyzed  for 
use  in  this  thesis.  Finally,  the  curl  operator  is  described  in  detail  for  comprehension 
of  the  this  operator’s  use. 

Chapter  HTI1  Methodology  details  the  theory  that  applies  to  Line  Of  Sight  (LOS) 
rate  equalization  and  the  looming  measurement.  It  provides  a  thorough  discussion  of 
LOS  rate  balancing  calculation,  looming  detection  measurement,  image  processing, 
optic  flow  development,  and  scene  development.  The  LOS  rate  balancing  discussion 
is  basic  theory  applied  in  two  dimensions  and  further  derives  the  balancing  control 
used  in  the  collision  obstacle  avoidance  algorithm.  Looming  detection  theory  in  this 
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research  is  related  to  the  line  of  sight  rate  equalization  and  assumes  one  contiguous 
obstacle  with  edges  as  features.  Optic  flow  is  developed  synthetically  using  a  method 
that  projects  a  surface  onto  a  unit  sphere.  Every  update  of  the  spherical  projection 
is  used  to  generate  the  changes  necessary  for  developing  optic  flow.  A  complement 
of  image  processing  techniques  are  implemented  to  obtain  optic  flow  vectors  that 
represent  obstacles.  The  scene  generation  and  trajectory  updates  are  discussed  in 
which  all  simulations  will  take  place  and  which  present  obstacles  for  the  vehicle  to 
navigate  around. 

Chapter  ITVl  details  the  method  of  testing  the  theory  developed  in  Chapter  Hill  A 
number  of  parameters  are  described  for  initiating  the  simulation;  scene  configurations, 
position  initialization,  resolution,  and  method  of  segmentation.  The  results  from  over 
920  simulation  runs  are  discussed.  The  successes  are  annotated  and  the  failures 
analyzed.  These  results  form  the  basis  for  the  demonstration  that  collision  avoidance 
can  be  attained  using  the  guidance  control  law  and  described  in  this  thesis. 

Chapter  |V]  presents  final  thoughts  on  the  work  is  presented  and  the  efficacy  of 
the  image  processing  for  accomplishing  the  goal  of  obstacle  avoidance.  The  chapter 
ends  by  setting  the  stage  for  future  work  and  aimed  at  increasing  the  capabilities  of 
this  research. 
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II.  Background 

This  chapter  introduces  the  concepts  and  theories  applied  in  this  research.  The  chap¬ 
ter  begins  with  a  thorough  discussion  of  optic  flow,  what  it  is  and  how  it  has  been  used 
in  the  literature.  It  details  information  on  biomimetic  research  and  how  it  influences 
this  thesis,  followed  by  a  short  background  on  the  concept  of  looming  and  its  origins. 
Discussion  on  collision  avoidance  for  different  vehicles  types  will  be  developed.  The 
final  concept  discussed  is  the  mathematical  definition  of  the  curl  operation. 

2. 1  Optic  Flow 

Optic  flow:  it  is  described  as  “the  speed  at  which  texture  moves  in  an  image 
focal  plane  as  a  result  of  relative  motion  between  the  observer  and  objects  in  the 
environment”  [24] .  It  can  give  valuable  information  about  the  arrangement  of  objects 
in  space  that  are  projected  onto  an  image  as  well  as  how  they  might  change  over 
time  m  The  optic  flow  generated  from  movement  is  the  result  of  two  distinct 
factors,  the  translational  velocity  and  the  angular  velocity  [22]  •  Tracking  the  changes 
or  shifts  in  the  images  using  these  two  components  can  give  information  about  motion 
of  the  camera  and  the  shape  of  the  environment  it  is  viewing  m  These  shifts  can  be 
visualized  by  using  a  quill  plot  of  the  velocity  vectors  and  overlaying  it  on  the  image. 
There  is  a  large  volume  of  research  in  this  area,  mainly  in  the  fields  of  robotics  and 
computer  vision  [20J  Optic  flow  has  also  been  studied  for  motion  estimation 

as  an  alternative  navigation  method  in  GPS  denied  environments. 

If  discontinuities  are  found  in  the  flow  field,  it  aids  the  ability  to  segment  regions 
of  interest  from  the  background  Ha-  Optic  flow  is  usually  measured  by  calculating 
velocity  vectors  in  some  field  of  view  in  an  image  -  see  Fig.  12.11 
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Figure  2.1:  Optic  Flow  in  a  basic  focal  plane  configuration  of  a  camera 

The  basic  premise  is  that  pixels  shift  from  one  frame  to  another,  given  that 
there  is  sufficient  texture.  These  shifts  can  be  calculated  using  the  following  formula 
from  [27] . 
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where  Sx  and  Sy  are  the  pixel  position  shifts,  Tx.Ty,  and  Tz  are  the  components  of  the 
spatial  translation  T  relative  to  the  camera,  x  and  y  are  the  corresponding  position 
vectors  and  a  feature  pixel  referenced  from  the  center  of  the  focal  plane,  Z  is  the  focal 
length,  and  wz,wy,  and  wx  are  the  angular  velocities  about  each  axis  as  described 

in  (5  and  [27J. 

According  to  [2_7],  the  optic  flow  is  generated  in  an  alternative  way  to  the  pre¬ 
viously  discussed  in  equation  (12 .11).  It  uses  a  correlation  method  in  which  (16x16 
pixel)  reference  blocks  ( RB )  of  an  image  are  compared  against  a  (32x32  pixel)  search 
window  (SW).  The  comparison  is  computed  using  a  Sum  of  Absolute  Differences 
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(SAD).  This  method  takes  the  difference  between  each  pixel  intensity  in  the  block 
and  the  search  window  and  then  sums  them  to  get  the  SAD  value.  The  block  is 
then  slid  across  the  search  window  generating  new  SAD  values  for  each  correlation 
position. 

7  7 

SAD(n,v)  =  EE  \Isw(v  +  b  u  +  k)  —  lRB(i,  &)|  (2.2) 

k=— 8  i=— 8 

SAD(Ax,Ay)  =  min^u)\SAD(iJb,v)\  (2.3) 

The  variables  Isw  and  Irb  are  the  intensities  of  the  search  window  pixels  and  the 
reference  block  pixels.  The  location  in  the  search  window  that  generates  the  minimum 
SAD  value  represents  the  optic  flow  vector  from  one  frame  to  another  E3 

The  Horn  and  Schunck  method  of  “Determining  Optic  Flow”  from  jT5]  finds 
the  optic  flow  based  on  a  brightness  constancy  idea.  The  idea  is  that  there  exists  a 
pattern  or  a  change  in  the  brightness  pattern  from  each  point  and  that  the  derivative 
of  the  brightness  where  E  is  the  brightness  constraint,  should  be  equal  to  zero. 
This  is  expanded  using  the  chain  rule  to 

dEdx  dEdy  dE  n 

!te~dt  +lhjdi  +  ~dt  ~  ^  ^ 

where 


dy_ 

dt 


(2.5) 

(2.6) 


The  single  linear  equation  becomes 


Exu  T  EyV  T  Ef  —  0 


(2.7) 
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where  Ex,  Ey ,  and  Et  are  the  partial  derivatives  of  the  brightness  function  E(x,  y). 
The  derivatives  are  then  estimated  spatiotemporally  to  hnd  the  optic  flow  using  mea¬ 
surements  from  an  8  block  cube,  where  4  blocks  are  from  a  previous  time  and  the 
same  4  blocks  are  at  the  current  time.  Next,  a  smoothing  solution  is  implemented 
that  attempts  to  eliminate  lighting  effects  such  as  reflections  or  areas  of  brightness 
that  can  cause  flow  vector  discontinuities.  This  is  done  by  minimizing  the  square  of 
the  magnitude  of  the  gradient  from  the  optic  flow,  as 


2  d2u  d2u 

v  u  =  — —  +  — — , 
oxz  oyz 

2  d2v  d2v 

dx2  dy 2 


(2.8) 

(2.9) 


This  smoothing  of  the  image  is  done  by  taking  a  local  pixel  average  brightness  and 
subtracting  the  brightness  for  that  pixel  from  the  average  brightness.  A  minimization 
method  is  used  to  find  the  least  sum  of  the  errors  for  the  rate  of  change  of  brightness 
or  the  departure  from  smoothness  in  the  optic  flow.  This  is  an  iterative  process  using 
the  Gauss-Siedel  method  for  each  point  in  the  image  na.  This  method  computes 
a  new  velocity  estimate  based  on  current  estimated  derivatives  and  the  average  of 
the  previous  velocity  estimates.  In  some  cases,  where  there  is  a  region  of  uniform 
brightness  and  the  derivatives  can  not  be  calculated,  the  region  is  simply  filled  in  by 
the  velocities  that  border  that  region  with  multiple  iterations. 

The  next  method  for  accomplishing  optic  flow  estimation  is  one  that  uses  a 
motion  estimation  and  another  spatiotemporal  method  but  with  a  least  squares  fit  [5] . 
This  method  makes  use  of  the  same  brightness  method  in  Equation  02.41)  from  [15j 
but  use  the  form 


Ax  =  b  (2.10) 

where  the  matrix  A  has  two  columns,  the  intensity  derivatives  in  x  and  y,  respectively. 
The  variable  x  is  the  velocity  vector  for  the  local  constant  velocity,  and  b  is  a  vector 
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consisting  of  the  derivatives  of  the  intensities  with  respect  to  time.  This  linear  matrix 
equation  is  then  used  to  create  lines  and  apply  the  Least  Median  Squares  (LMS)  to 
find  the  best  fit  for  the  motion.  This  is  accomplished  by  defining  the  set  of  lines  as 

y  =  rriiX  +  n*,  i  =  1,2, . ,p  (2.11) 

where  m  and  n  are  constants  for  each  line  of  the  set  of  p  lines.  The  closest  point  (CP) 
to  the  given  pixel  location  is  found,  where  the  CP  is  the  point  that  has  the  minimum 
error  of  the  vertical  distances  to  the  set  of  lines  and  described  as 

ECP=  -  y  f-  (2-12) 

2=1 

The  standard  regression  (SR)  error  is  calculated  as 

ESR  =  ™l,n)J2(mXi  +  n~yrf-  (2‘13) 

2=1 

The  claim  is  made  that  Esr  can  be  transformed  into  Ecp  by  using  the  (m,n)  resulting 
in  the  minimum  and  using  the  inverse  transform  of  Equation  (12.111)  and  solving  for 
(x,y).  This  new  point  is  the  new  position  found  for  the  original  pixel  and  a  velocity 
vector  can  be  resolved  and  reaccomplished  for  each  pixel  to  develop  the  optic  flow 
field  [3]. 

The  subsequent  method  analyzes  optic  flow  in  a  camera’s  focal  plane  and  infers 
information  about  the  motion  of  the  camera  itself  as  well  as  information  about  the 
shape  of  the  environment  it  is  observing  [22] .  This  is  key  since  it  describes  how  optic 
flow  is  generated  in  simulations  presented  later  in  this  thesis.  The  optic  flow  fields  used 
in  this  research  were  based  on  this  interpretation.  The  camera  with  its  focal  plane 
is  modeled  as  a  projection  onto  a  sphere  with  a  single  projection  point  originating 
from  the  center  of  the  sphere.  From  Fig.  12.21  Qi  are  multiple  projection  point  unit 
vectors,  or  resolution  vectors,  where  the  ith  index  is  the  number  of  resolution  vectors 
extending  from  Q.  These  vectors  extend  to  the  top  and  bottom  of  the  surface  and 
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Figure  2.2:  Optical  Flow  on  a  Spherical  Projection  Surface  (reprint  from  [22]) 

multiple  points  inbetween.  P,  is  the  range  along  each  Qt  projection  vector  to  the 
surface  projection,  and  is  a  Euclidean  vector  norm,  V  is  simply  the  translational 
velocity  vector  of  the  surface,  and  Q  is  the  angular  velocity  vector  of  the  spherical 
projection  surface.  The  spherical  projection  surface  is  analogous  to  the  location  of 
the  camera’s  pinhole.  In  [22],  Qif  is  the  rate  of  change  of  the  projection  vectors  onto 
the  unit  sphere  resulting  in  optic  flow  and  can  be  calculated  according  to 

[£-(£•4)41 

4  = - ~  -  +  0  x  Qi  where(l  <  i  <  n).  (2-14) 

Pi 

In  this  example,  all  of  the  vectors  are  in  3-D  Cartesian  space.  In  Equation  02.1411. 
n  directly  relates  to  the  number  of  projection  vectors  and  is  the  resolution  of  the 
projection.  This  work  in  [22]  further  describes  a  least  squares  fit  to  determine  the 
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camera’s  motion  from  the  computed  optic  flow  field.  As  noted  previously,  the  work 
in  [22]  serves  as  a  basis  for  generating  optic  flow  fields  in  the  simulations  in  this  thesis. 

2.2  Biomimetics 

Biomimetics  is  the  study  aimed  at  understanding  the  mechanics  of  biological 
systems  and  their  neurological  behavior  M-  It  encompasses  a  wide  variety  of  ani¬ 
mals  from  insects,  to  primates.  The  goal  is  to  better  understand  the  mechanisms  that 
animals  use  to  navigate  through  their  environment  and  exploit  that  information  with 
technology  to  create  autonomous  vehicles.  Much  of  what  is  known  today  about  optic 
flow  (Section  12.11)  comes  from  the  study  of  insects  [20] .  A  significant  amount  of  re¬ 
search  on  the  visual  system  of  the  fly  has  been  accomplished  to  gain  an  understanding 
of  how  they  use  vision  for  collision  avoidance  [TO] ,  [TT] ,  [T9] ,  and  [T2] .  Flies  are  used 
because  their  neuro-pathway  has  been  studied  extensively  and  is  well  known  [TU] .  Of¬ 
ten  Blowflies  are  used  due  to  its  size  US-  The  information  learned  from  flies  provides 
the  crux  of  aircraft  motion  estimation,  but  there  is  also  a  lot  of  insight  about  collision 
avoidance  from  optic  flow.  This  is  because  the  fly  uses  Elementary  Motion  Detection 
(EMD)  for  determining  motion  in  it’s  visual  system.  EMD  is  the  basic  process  for 
determining  optic  flow  across  its  vision  field  mi  and  [19] .  Thus  optic  flow  shows  some 
promise  for  accomplishing  collision  avoidance. 

The  study  of  the  Blowfly  and  its  visual  system  in  inn  yielded  some  interesting 
results.  It  is  discerned  that  in  the  fly  and  other  moving  animals  that, “optic  flow  is  an 
important  source  of  information  about  self  motion  and  the  three  dimensional  layout  of 
the  environment”  [10] .  The  research  discusses  four  key  topics;  unique  organization  of 
the  retinotopic  input  in  the  neurons  that  process  optic  flow,  combination  of  optic  flow 
in  both  eyes,  the  accuracy  of  the  optic  flow  information  being  processed,  and  the  per¬ 
formance  of  the  neurons  under  motion  stimulus.  The  fly’s  eye  is  built  with  numerous 
facets  of  ommatidia,  which  are  thousands  of  small  lenses  that  make  up  the  compound 
eye  and  focus  the  light  to  the  photo  receptor  nerves.  They  are  arranged  in  horizon- 
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tal  and  vertical  rows  that  give  it  a  panoramic  view  of  nearly  360°.  An  interesting 
thing  to  note  is  that  the  fly’s  eye  contains  multiple  sets  of  neurons  that  are  sensitive 
to  “preferred  directions”  m  The  preferred  direction  simply  means  the  direction  of 
optic  flow  that  the  neuron  seems  to  create  a  strong  stimulus  from.  These  neurons 
are  located  in  specific  areas  of  the  eye  to  detect  the  preferred  direction  most  likely 
occurring  in  the  eye  at  its  location.  Two  specific  neuron  classes  are  the  Horizontal 
System  (HS)  and  the  Vertical  System  (VS).  HS  primarily  detects  horizontal  motion 
and  VS  detects  vertical  motion.  The  fly  uses  both  eyes  for  self  motion  estimation. 
The  large  fields  of  view  for  each  eye  can  still  be  fooled  by  optic  flow  patterns.  For 
ambiguous  optic  flow  patterns,  it  is  the  fusion  of  the  information  from  both  eyes  that 
resolve  the  ambiguities.  For  example,  when  translation  occurs,  it  creates  an  optic 
flow  field  that  extends  across  the  entire  field  of  view  in  a  single  direction  for  each 
eye.  When  rotating,  the  optic  flow  field  also  extends  across  the  entire  field,  except 
that  each  eye  observes  an  opposite  direction  of  flow.  This  difference  allows  the  fly  to 
discriminate  between  translation  and  rotation  which  is  difficult  to  accomplish  with 
one  eye.  The  procedure  in  which  the  gauged  response  to  motion  stimulus  is  measured 
can  be  considered  noisy.  This  is  because  the  cell  membrane  tends  to  carry  electrical 
potentials  that  can  create  various  responses  to  the  same  stimulus.  It  is  stated  that 
large  motion  and  rapid  changes  will  create  a  favorable  condition  for  measurement  to 
the  membranes  potential.  The  idea  is  that  the  potential  in  the  neurons  membrane 
spike  and  remain  spiked  while  the  motion  is  occurring,  but  when  slow  changing  motion 
is  observed,  this  potential  is  smaller  and  harder  to  discern  from  normal  fluctuations. 
Thus,  it  is  believed  that  with  the  correct  stimulus,  the  researchers  can  identify  re¬ 
sponses  to  observed  motion.  Finally,  reactions  to  the  neurons  from  a  natural  setting, 
like  in  free  flight  or  walking,  is  unknown.  This  means  that  the  measurements  and 
experiments  may  not  always  depict  what  the  animal’s  self  motion  might  be  in  this 
natural  situation.  Instead,  special  cases  have  been  used,  for  example,  tethering  the 
animal  in  a  flight  simulator  and  recording  the  dynamic  inputs  to  the  brain  from  the 
vision  system  for  reenactment  of  natural  flight.  This  is  done  to  observe  the  suggested 
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response.  This  seems  to  show  that  a  series  of  inputs  can  elicit  a  response  to  turn 
the  animal  around  its  vertical  axis,  suggesting  that  optic  flow  inputs  to  the  eye  are 
exploited  for  this  means.  This  research  supports  the  idea  that  optic  flow  can  be  used 
in  video  sequences  to  elicit  a  response  for  control  of  a  small  UAV  or  MAV. 

The  discussion  based  on  [19j  pertains  to  the  study  of  the  Blowfly.  The  authors 
further  identify  major  neurons  that  are  sensitive  to  a  “local  preferred  direction”  (LPD) 
which  is  the  “preferred  direction”  described  in  Pi-  This  paper  describes  the  HS  and 
VS  neurons  as  horizontally  and  vertically  sensitive  respectively  -  see  Fig  12 .81  Here,  HI 


direction  of  motion  [  °  ] 


direction  of  motion  [  °  ] 


Figure  2.3:  Neuron  Response  to  Directional  Motion  (reprinted  from  |19jf 
pertains  to  HS  neurons  and  VI  pertains  to  VS  neurons  where  the  plot  is  of  electrical 
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spikes  in  the  neuron  for  direction  motion  displayed  to  a  flies  eye.  This  was  accom¬ 
plished  by  a  visual  stimulus  of  a  small  black  dot  put  in  the  flies  view  rotated  around  in 
a  circle  to  get  full  360°  degree  rotation  for  all  motion  directions.  From  this,  it  is  clear 
that  the  strongest  responses  in  the  neurons  occur  at  some  LPD.  The  main  distinction 
of  this  research  is  that  the  stimuli  is  further  investigated  and  creates  a  response  in 
the  neuron.  The  speed  at  which  the  dot  was  rotated,  the  size,  shape,  and  contrast  of 
the  dot,  and  even  the  diameter  of  the  rotation  of  the  dot  is  changed.  It  is  noted  that 
when  the  speed  of  the  dot  rotation  is  changed  that  the  LPD  remains  constant,  but  as 
the  speed  of  the  rotation  increases  the  spike  in  the  neuron  potential  increases  as  well. 
Next,  it  is  discovered  that  size,  shape  and  reversal  of  the  contrast  for  the  dot  made  no 
difference  for  the  LPD,  even  as  the  dot  is  varied  in  rotation  speed.  Finally,  the  size 
of  rotation  did  not  change  the  LPD  except  for  very  small  rotation  diameters.  The 
hypothesize  is  that  when  the  diameter  is  small  there  is  considerable  overlap  of  the  dot, 
creating  minimal  motion  detection  changes.  It  is  further  stated  that  the  ommatidia, 
or  the  compound  eye  facets,  offset  enough  that  the  small  rotation  might  not  engage 
motion  detection  in  neighboring  ommatidia,  and  by  extension,  the  HS  neuron.  This 
provides  more  evidence  that  sensitivity  to  vertical  and  horizontal  queues  separately 
might  be  a  simple  solution  to  3-D  motion  analysis  and  collision  avoidance.  Obviously, 
the  fly  fuses  information  from  thousands  of  neurons  from  various  locations  in  the 
eye  to  make  decisions,  but  local  motion  is  accomplished  across  small  segments  of  the 
eye  which  might  be  more  analogous  to  the  single  camera  model  employed  in  this  thesis. 

The  work  in  im  and  [12],  explores  how  the  locust’s  Lobula  Giant  Movement 
Detector  (LGMD),  which  is  a  neuron  in  its  optic  lobe,  responds  to  approaching  objects 
that  are  on  a  collision  course  with  the  locust.  They  investigate  the  reactions  of  this 
nerve  to  looming  stimuli.  The  stimuli  is  generated  using  a  black  dot  or  square  on  a 
bright  background  and  reenacting  a  collision  event  by  moving  the  object  closer  to  the 
eye  of  the  locust  while  perpendicular  to  the  eye  -  see  Fig.  I2.4L4.  The  nerve  responds 
with  an  increased  firing  rate  that  peaks  and  then  falls  off.  The  stimulus  object’s  half 
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size  is  l,  and  v  is  the  approaching  objects  velocity  which  is  used  to  determine  the 
angular  size  «di  ra  -  see  Fig.  12.41-  where  t  denotes  the  time-to-collision 


9(t)  =  2tan  1  (— ). 


(2.15) 


Figure  2.4:  A:  perpendicual  object  approaching  an  eye;  B:  The  plot  of  Time  to  Col¬ 
lision  with  Looming  Component;  C:  The  Firing  Rate  of  the  LGMD  (figure  reprinted 
from  |12j) 


The  convention  used  here  is  that  v  <  0  for  an  approaching  object  where  the 
speed  is  used  as  the  absolute  value,  (|u|  >  0).  In  locusts,  the  eyes  are  set  on  opposite 
sides  of  the  head  and  there  is  less  than  20°  overlap  in  the  binocular  vision  m •  This 
means  that,  “because  the  receptive  field  is  monocular  and  each  eye  views  largely 
independent  portions  of  visual  space,  9{t)  fully  describes  the  time  course  of  retinal 
stimulation  by  the  approaching  object”  HU.  It  is  then  stated  that  according  to 
Equation  (12.151).  9{t)  and  Bit)  are  nonlinear  functions  and  that  the  temporal  changes 
in  each  occur  due  to  the  ratio  of  ^ .  The  looming  ratio  ^  is  varied  and  the  responses 
are  tracked  to  show  that  the  peak  firing  times  occur  more  often  toward  the  collision 
time  when  the  ratio  decreases.  This  means  that  the  nerve  is  sensitive  to  the  loomin 
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ratio  above,  and  is  interpreted  to  mean  that  either  the  object  is  farther  away  or  moving 
slowly,  suggesting  that  a  collision  is  not  imminent.  This  is  further  supported  by  the 
apparent  linear  relationship  between  the  peak  firing  time  and  the  ratio  From 
these  results,  it  appears  that  the  threshold  value  for  peak  firing  is  when  6  is  between 
15°  —  35°.  Additionally,  the  peak  firing  times  appear  to  be  independent  of  luminance, 
contrast,  body  temperature,  variations  in  shape,  texture,  and  approach  angle.  This 
work  provides  the  foundation  for  looming  measurement.  The  idea  of  angular  velocity 
of  an  approaching  obstacle  as  the  basis  for  looming  detection  is  similar  to  the  looming 
calculation  used  in  this  thesis. 


2. 3  Looming 

Looming  in  vision  is  a  stimulus  that  animals  and  humans  use  for  collision  avoid¬ 
ance  HB|.  Looming,  which  is  related  to  the  expansion  of  an  object  in  an  animal’s  or  a 
camera’s  focal  plane,  can  be  calculated  from  optical  flow  or  determined  by  flow  field 
divergence  [18] .  This  divergence  will  arise  from  nonzero  looming  acquired  from  trans¬ 
lation  along  the  observer’s  optical  axis  [6].  This  key  indicator  of  motion  is  essential 
for  many  visual  controlled  functions  [BJ.  In  most  cases,  using  optic  flow  for  transla¬ 
tion  estimation  can  be  assumed  to  be  local  motion,  but  divergence  can  be  assumed  to 
occur  across  the  entire  image  [6j .  Looming  is  an  imperative  visual  cue  used  in  nature 
and  it  can  be  identified  in  images  and  exploited  for  collision  avoidance. 

The  author  ra,  measure  looming  in  a  scene  by  studying  the  texture  and  how  it 
changes  temporally.  The  looming  can  then  be  calculated  by  the  change  in  this  texture 
relative  to  the  previously  found  texture.  They  also  can  determine  basic  orientation 
of  the  surface  from  a  set  of  “one  dimensional  directional  densities  of  the  texture 
primitives”  m-  Looming,  L,  is  described  as 


L  = 


dR/dt 

R 


(2.16) 
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where  R  is  the  range  to  the  center  of  an  obstacle.  The  authors  further  state,  “Looming 
is  independent  of  rotation  of  the  camera  and  it  can  be  conveniently  measured  using 
a  logarithmic  retina”  ra-  They  also  indicate  that  looming  is  related  to  r,  the  time- 
to-collision,  also  described  in  PJ.  Although  looming  is  related  to  r,  it  is  not  identical. 
Confirmation  from  other  references  state  that  the  rate  an  object  expands  in  an  image 
is  proportional  to  the  looming  value.  Hence,  an  equation  is  formed  that  calculates 
the  expansion  of  texture  in  a  scene  by  first  using  the  relationship 


r 


2 


7  PC 
k 


(2.17) 


where  r  is  the  range  from  the  camera  pinhole  to  the  feature  in  the  image  plane,  /  is 
the  focal  length,  7  and  k  are  described  as  the  textural  densities,  and  C  is  defined  as 
cos  6  -  see  Figs.  12.51  and  12.61  Differentiating  both  sides  and  divide  by  2 r2  yields 

dr/dt  1  d'y/dt  1  dC/dt 

r  ~  2  7  ^  2  C  <2'18) 

Simplifying  and  using  Equation  (12.171).  the  looming  equation  then  becomes 


■4- - 

y 


Figure  2.5:  Image  depicts  the  orientation  of  the  camera  and  the  surface  observed. 
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Figure  2.6:  Diagram  displays  the  features  projected  on  a  focal  plane 


(2.19) 


Equation  (12.191]  is  used  to  determine  the  expansion  of  textures  which  indicates  loom¬ 


ing.  They  state  that  can  |)C  measured  directly  from  the  texture  density  in  the 
image  but  6  and  ^  are  calculated  much  differently.  As  noted  previously,  9  is  computed 
from  the  texture  density  along  several  lines  which  provides  a  look  up  table  used  for 
the  texture  density.  Then  ^  is  calculated  as  9  k  —  9k- a t  where  At  is  an  arbitrary  time 


step,  but  in  general,  this  time  step  would  increase  as  velocity  increases.  This  method 
for  detecting  looming  requires  decisions  based  on  a  priori  knowledge,  for  example, 
the  look  up  table  of  9  values.  In  a  real-time  UAV  collision  avoidance  system,  this 
information  is  not  privy.  For  this  thesis,  the  texture  density  tracking  idea  is  put  into 
use  but  by  using  edges  as  the  texture  information  and  tracking  them  temporally,  and 
using  LOS  angles  that  can  be  measured  in  the  focal  plane. 

The  authors  in  [211]  uses  flow  field  divergence  for  exactly  the  opposite  purpose 
that  is  being  researched  in  this  thesis.  The  use  of  flow  fields  is  interpreted  to  control 
the  docking  of  mobile  robots  into  docking  stations.  The  main  idea  is  that  as  a  mobile 
robot  approaches  an  object,  it  needs  to  slow  down  and  position  itself  correctly.  It 
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is  envisioned  that  this  could  be  used  for  robot  automation  to  complete  tasks  like 
redocking  in  charging  stations,  pallet  lifting,  or  transportation  of  goods  [23].  The 
research  intends  to  accomplish  this  by  using  optic  flow  to  determine  looming  from 
diverging  flow  vectors  and  determining  the  focus  of  expansion  (FOE)  location  and 
time  to  collision.  The  time  to  collision  is  directly  calculated  from  the  diverging  flow 
field  or  looming  vectors.  The  problem  is  that  accurately  describing  the  divergence  is 
difficult,  given  rotational  components  of  the  optic  flow.  In  this  work,  it  is  intended 
to  mitigate  the  problem  caused  by  rotational  components  and  detail  how  the  actual 
FOE  is  found.  First  begin  by  defining  the  divergence  D  as 


D  UX  Vy 


(2.20) 


where  ux  and  vy  are  the  partial  derivatives  of  horizontal  and  vertical  components  of 
optic  flow  respectively.  The  time  to  collision  r  is  then  defined  as 


Z  _  2 
¥r~D' 


(2.21) 


where  Z  is  the  distance  to  the  object  along  the  translational  axis  and  Tr  is  its  velocity. 
From  here,  there  is  a  need  to  compensate  for  the  rotational  components  in  the  flow 
held  and  is  accomplished  as  follows  -  see  Fig.  12.71  First,  it  begins  by  defining  the 


Zo 


Figure  2.7:  Geometric  Configuration 
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distance  to  the  object  as 


*(*■»>  <2-22> 

Jx  Jy 

Z0  being  the  distance  to  the  surface  of  the  object  along  the  optical  axis,  x  and  y  are 
image  coordinates,  and  fx  and  fy  are  the  focal  lengths  expressed  in  pixels.  Transla¬ 
tional  velocity,  Tc,  and  angular  velocity,  uc,  of  the  camera  are  defined  by  -  see  Fig. 

EZ3 


Tc  =  [o  Tr  j3Tr  7  Tr\ , 


=  [“ 


UJr  =  \U). 


(Jy  UJ. 


]■ 


(2.23) 

(2.24) 


Then  using  these  equations  for  optic  flow 


u(x,  y )  =  fx 
v(x,y)  =  fy 


fMjtzff)  .  fy 

Z(x,y )  x  fxfy 


/  X  \  u 

~UJy{1  +  -p)  +0JzYz 


Z(x,y) 


+  UJX(1  +  J2)  ~ 

J  V 


(J, 


xy  x 

y  ~T~f  xiz  — 
JxJy  Jx 


(2.25) 

(2.26) 


and  describing  the  distance  to  the  FOE  point  in  the  image  at  (; x',y' ),  or  Z(x,y ) 
becomes 


Z(x,y)  = 


Z(x',y') 


1  -a^-b^Y 

Jx  Jy 


(2.27) 
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Substituting  Z  into  the  optic  flow  Equations  (12.251)  and  (12. 2 til)  yields. 


u(x,y ) 


v{x,y ) 


7 Tr(x  -  fxa ) 

1 - 

I 

-O 

1 

1 

1 

T — 

1 _ 

Z(x',y') 

T  uj. 

7 Tr{x  -  fxa) 

fx  fy 

xy  .  x2 .  y 

x  —p - ^ y(fx  +  ~r)  +  Uz  —  . 

Jy  Jx  Jx 

a( x  -  x')  b(y  -  y’) 

Z(x',y') 

fx  fy 

.  .  y\  xy  x 

"b^-h \fy  T  ~jr )  —  ^z~jr- 

Jy  Jx  Jx 


(2.28) 


(2.29) 


The  partial  derivatives  are  taken  of  u(x,y )  and  v(x,y)  and  substituting  x  =  x'  and 
y  =  y'  because  it  is  assumed  that  the  optical  flow  at  the  FOE  is  zero,  this  leads  to 


u. , 


7  Tr 


foe  Z(x',y') 

=  jTr 
foe  Z (x1,  y') 


,  X 


1  —  a(— — \-  a) 
Jx 

+p) 

Jy 


y'  2xr 

T  ^x~z~  ^ y—f  ? 

Jy  Jx 

2  y'  x' 

T  ^x  (  ~Z~  )  —  My  ~T  • 
Jy  Jx 


Taking  the  sum  of  these  as  in  Equation  (12.201).  develops 


Df  = 


7  Tr 


Z(x',y') 


,x'  ,  ,  .  y' 

a(— — f  a)  +  b(— — |-  (3  —  2 

Jx  Jy 


!  Q  f^xU  UJyX  ^ 

+  ~r> 

Jy  J  x 


(2.30) 

(2.31) 


(2.32) 


where  Df  is  the  divergence  at  the  FOE  and  with  some  type  of  inertial  measurement 
unit  Tr,  lux,  and  cuy  can  be  easily  determined.  The  authors  continue  on  to  calculate  the 
time  to  collision  for  the  FOE  and  imply  that  due  to  the  use  of  a  ground  robot  the  u jz 
can  be  neglected.  Furthermore,  the  researchers  use  this  time-to-collision  variable  to 
establish  a  feedback  control  to  decrease  the  robot’s  velocity  as  an  object  approaches. 
Knowing  that  looming  is  a  key  calculation  for  this  thesis  and  if  the  time  to  collision 
were  known  as  in  this  work,  a  control  could  be  established  to  avoid  the  obstacle  versus 
decreasing  the  velocity. 
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2-4  Collision  Avoidance 


The  authors  in  [8] ,  use  a  simulated  ground  robot  with  a  simulated  point  camera 
and  a  held  of  view  of  0°  —  330°.  Optic  how  is  generated  from  translational  motion 
only.  This  is  done  because  the  simulation  of  the  robot  motion  is  in  a  “piecewise  linear 
manner”  [8].  The  Optic  how  is  then  calculated  by 


a  =  |fo|sin/? 
d 


(2.33) 


where  $  is  the  angular  velocity  of  the  optic  how,  h  is  the  robots  velocity,  ft  is  the 
angle  to  a  specihc  feature  used  to  calculate  optic  how,  d  is  the  distance  between  the 
camera’s  simulated  point  to  the  feature.  The  research  also  describes  the  time-to- 
collision  variable  r,  where  r  =  |.  If  it  falls  below  a  threshold,  the  robot  turns  180°. 
The  r  value  is  similar  to  a  looming  detection  value.  The  idea  is  to  navigate  through 
a  maze  by  equalizing  the  optic  how  seen  on  each  side  of  the  robot  with  set  of  two 
cameras.  The  optic  how  rates,  / 3l  and  (3r  for  the  left  and  right  side  respectively,  are 
calculated  and  averaged.  The  robot  turns  according  to  the  control  rjMi  given  by 


f'bal  —  K  (Pl  ~  Pr) 


(2.34) 


where  k  is  a  scaling  constant /gain.  The  basic  concept  is  that  when  the  average  optic 
how  is  higher  on  one  side  than  the  other,  the  robot  turns  in  the  away  from  the  higher 
optic  how.  The  paper  continues  to  discuss  varying  methods  to  attain  collision  avoid¬ 
ance  through  a  maze  while  hnding  a  goal  location.  The  basic  Equation  (12.241)  already 
presented  is  modihed  to  account  for  some  a  priori  information.  Using  Equation  (12.341) 
information  it  makes  decisions  on  turns  and  adjusts  a  bias  that  controls  the  robot  to 
remain  closer  to  one  wall  as  compared  to  the  other  during  the  turn.  The  research 
furthermore  includes  a  technique  where  salient  values  are  given  at  specihc  location 
to  change  the  control  bias  to  match  the  circumstance.  In  this  thesis,  there  is  no  pre¬ 
sumption  of  a  goal  location,  so  the  idea  of  balancing  the  optic  how  and  providing  a 
control  mechanism  is  a  major  take  away  from  this  work.  The  rest  of  the  work,  while 
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interesting,  does  not  have  much  importance  for  collision  avoidance  and  requires  too 
much  a  prior  information. 

2. 5  Curl  Operator 

The  curl  operator  is  a  vector  operator  that  is  used  to  determine  if  a  rotation 
exists  in  a  particular  vector  field  EB  Imagine  a  paddle  wheel  placed  into  the  currents 
of  a  river  -  see  Fig.  12.81  The  paddle  wheel  will  rotate  depending  on  the  vector  strength. 
If  the  magnitude  of  the  vectors  is  equal  there  is  no  curl  (i.e.,  no  rotation),  but  if  the 
magnitude  is  uneven  then  a  curl  is  evident  at  between  the  two  vectors.  Using  the 
right  hand  rule,  positive  curl  is  a  rotation  that  is  counterclockwise  and  negative  curl 
is  clockwise  rotation  [21]  -  see  Fig.  12.81 


Figure  2.8:  Description  of  Curl  in  a  Vector  Field 

The  curl  operator  is  defined  as  EH, 

,  .  _  .  ,  un  <f  A  ■  dl 

curl  A  =  Vx4  =  lim  — - . 

As^O  As 


(2.35) 
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From  Equation  (12.851).  it  can  be  seen  that  it  can  be  equivalently  calculated  by  [2T] 


curlA  = 
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dx 
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(2.36) 


)  (2.37) 


The  math  shows  that  the  curl  is  always  orthogonal  to  the  plane  in  which  the  the 
vectors  reside.  In  this  thesis,  the  optic  flow  field  is  described  as  x  and  y  vectors  only, 
hence  there  will  only  be  curl  values  in  the  z  direction. 


2. 6  Summary 

There  is  key  information  that  is  derived  from  research  already  accomplished. 
The  application  of  optic  flow  is  necessary  to  generate  measurements  of  possible  ob¬ 
stacles.  The  biomimetics  display  that  optic  flow  can  be  used  for  collision  avoidance. 
The  looming  theory  sets  the  ability  to  accomplish  measurement  of  looming  from  optic 
flow.  Collusion  avoidance  is  used  in  various  methods  in  the  literature  and  the  method 
described  in  this  chapter  is  applied  in  this  work.  The  thesis  combines  multiple  aspects 
from  these  research  areas  to  creat  a  single  obstacle  collision  avoidance  algorithm. 

The  theory  represented  in  the  following  chapter  is  based  on  five  main  topics: 
optic  flow,  biomimetics,  looming  detection,  collision  avoidance,  and  the  curl  operator. 
All  of  these  concepts  need  to  be  well  understood  for  completing  the  work  accomplished 
in  this  thesis.  In  the  next  chapter,  the  theories  will  be  presented  and  shown  how  they 
are  applied. 
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III.  Methodology 


3. 1  Introduction 

Flying  Micro  Aerial  Vehicles  (MAVs)  in  unstructured  environments  requires  the 
ability  to  autonomously  avoid  obstacles  [25].  The  MAV  must  be  able  to  avoid  the 
obstacles  when  their  positions  are  not  known  ahead  of  time,  and  without  operator 
intervention. 

The  measurement  of  optical  flow  can  be  used  for  autonomous  navigation  [26] 
and/or  obstacle  avoidance  guidance  [8j.  Motivated  by  biomimetics,  an  autonomous 
guidance  scheme  for  obstacle  avoidance  has  been  suggested  -  see  for  example  [IB] .  The 
proposed  guidance  scheme  relies  solely  on  nullifying  the  difference  of  the  optical  flow 
generated  by  obstacles  in  the  field  of  view  of  a  single  forward  omni  directional  camera. 
Nullifying  the  difference  of  the  Line  Of  Sight  (LOS)  rates  generated  by  obstacles  in 
the  field  of  view  of  the  camera  is  also  an  interesting  guidance  concept,  in  particular, 
since  the  direct  measurement  of  LOS  rates  is  easily  implemented,  e.  g.,  in  Infrared 
(IR)  guided  air-to-air  missiles. 

Biomimetic  inspired  guidance  concepts  are  intuitive.  Consider  the  simplest  case 
where  there  are  just  two  obstacles  and  the  MAV  is  initially  located  on,  and  aligned 
with,  the  orthogonal  bisector  of  the  segment  connecting  the  two  obstacles  -  see  Fig. 
ED  The  proposed  guidance  laws  will  generate  a  flight  path  that  keeps  the  UAV  on 
the  orthogonal  bisector  such  that  the  MAV  flies  between  the  obstacles.  However,  this 
is  a  special  case.  For  all  other  cases,  where  the  MAV  is  initially  off  the  orthogonal 
bisector  and/or  is  not  aligned  with  the  orthogonal  bisector,  the  optic  flow  balancing 
guidance  law  will  not  steer  the  MAV  toward  the  orthogonal  bisector.  We  will  show 
that,  contrary  to  one’s  intuition,  the  proposed  guidance  method  might  result  in  a 
collision  with  an  obstacle  when  the  literal  line  of  sight  rates  or  optic  flow  rates’ 
difference  are  nulled  for  a  360°  FOV  camera.  For  this  case,  the  proposed  guidance 
method  will  always  result  in  a  collision  with  an  obstacle  when  the  difference  of  the  LOS 
rates  is  nulled;  however,  by  using  optic  flow  balancing  guidance  with  realistic  sensor 
restrictions,  obstacle  avoidance  can  be  achieved.  The  stability  of  the  ideal  trajectory, 
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where  a  MAV  is  initially  on  the  orthogonal  bisector  of  the  segment  connecting  the 
two  obstacles,  is  also  analyzed.  It  is  shown  that  the  control  law  is  less  stable  the 
closer  the  air  vehicle  gets  to  the  objects.  Finally,  the  looming  concept  is  introduced. 
Looming  in  the  2-D  scenario  is  quantified  as  a  LOS  rate  to  the  obstacle’s  features  in 
its  FOV.  It  is  shown  that  the  obstacle  can  be  avoided  with  a  control  law  that  strives 
to  take  advantage  of  FOV  restrictions. 

3.2  Analysis  1  -  Two  Cameras 

For  simplicity,  we  initially  consider  a  two-dimensional  scenario  where  the  motion 
is  constrained  to  the  Euclidean  plane  with  two  fixed  obstacles  separately  tracked 
by  two  electro-optical  sensors.  The  MAV  is  initially  at  point  A  and  the  two  fixed 
obstacles  are  at  points  B  and  C  separated  by  distance  d.  The  objective  is  to  steer 
the  air  vehicle  between  the  obstacles.  The  range  of  the  MAV  from  B  is  r  and  the 
angle  included  between  the  baseline  \BC\  and  the  radial  from  B  to  the  MAV  at  A  is 
6.  The  “engagement  triangle”  ABC  is  shown  in  Figure  I  VI  l  The  speed  of  the  MAV, 
V,  is  constant  and  the  relative  course  of  the  MAV  is  0,  as  shown  in  Figure  IH7T1 


A 


Figure  3.1:  Problem  geometry  for  MAV  at  A  avoiding  obstacles  B  and  C 

The  optical  flow  in  the  cameras’  focal  planes  generated  by  obstacles  B  and 
C,  respectively,  causes  the  LOS  to  B  and  C  to  rotate.  The  rates  of  the  LOS  to 
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B  and  C  are  measured.  A  biomimetic  inspired  obstacle  avoidance  guidance  law  - 
see  Equation  (13.311  -  relies  on  nulling  the  difference  between  the  rates  of  the  LOS  to 
obstacles  B  and  C.  The  MAV’s  course  angle  <p  is  set  accordingly. 

To  derive  the  differential  equation  governing  the  MAV’s  motion,  the  rate  of  the 
LOS  to  obstacles  B  and  C  is  calculated  in  Equations  (13.111  and  (13.2ft.  respectively: 


V  sirup 


(3.1) 


0  = 


r 


and 


(3.2) 


where  R  is  the  range  to  obstacle  C  and  ip  is  the  angle  included  between  the  LOSs  AC 


and  l-BC*!.  The  course/control  (p  of  the  MAY  is  determined  by  the  guidance  law 


9  =  ip 


(3.3) 


and  from  Equations  (13.  lH  and  (13.211  we  obtain 


sin 


sin(ZA  —  <p ) 


r 


R 


sin(ZA)  cos  <p  —  cos(ZA)  sin  <p 


R 


which  yields  the  control 


tan d>  = - .  ,  .. - — —  . 

cos(ZA)  +  R/r 


sin(ZA) 


(3.4) 


Next  from  A  ABC  we  calculate 


(3.5) 


(3.6) 
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and 


R  =  Vr2  +  d2  —  2 dr  cos  9  . 


(3.7) 


Inserting  Equations  (I3.5D-03.7I)  into  Equation  (I3.4D  yields  the  control  law 


tan</>  = 


dr  sin  d 


2  r2  +  d2  —  3dr  cos  6* 


(3.8) 


Evidently,  due  to  obstacle  limitations,  the  relative  course  4>  of  the  MAY  must  satisfy 


0  <  </>  <  ZA 


(3.9) 


We  will  describe  the  motion  of  the  MAY  using  polar  coordinates  r,  d  where 


r  =  —  V  cos  </>,  (3.10) 

and 

V 

9  = - sin0  .  (3-11) 

r 

The  quotient  of  the  preceding  equations  yields 

(19  1 

—  =  -  tan0  .  (3-12) 

dr  r 

Combining  Equations  (13.8D  and  (13.1 2D.  and  using  the  non-dimensional  range  r  — >  r/d, 
gives  the  differential  equation 


dd 

dr 


sind 

1  +  2r2  —  3r  cos  d 


6(r0)  =  90  . 


(3.13) 


There  are  two  axes  of  symmetry  in  the  solution  of  the  differential  Equation 
(13.1 3D.  These  consist  of  1)  the  extension  of  the  segment  | D C \ ,  and,  2)  the  orthogonal 
bisector  of  the  segment  \BC\.  Note  that  the  non-dimensional  separation  \BC\  is  now 
1.  By  symmetry,  the  initial  conditions  of  interest  satisfy  0<do<7r/2,  1/2  <  /-0  -  For 
a  path  beginning  on  the  orthogonal  bisector  between  points  B  and  C  in  Figure  13.11 
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the  initial  conditions  satisfy 


6*o  =  arccos(- — )  . 

2ro 

The  solution  of  the  differential  Equation  (18.11)  then  satisfies 

9  =  arccosf— )  . 

v2  r’ 

In  this  special  case,  the  MAV’s  course  is  orthogonal  to  the  segment  \BC\  and  the 
MAV  will  stay  on  the  orthogonal  bisector  of  the  segment  \BC\  and  will  pass  between 
the  obstacles  B  and  C .  This  conforms  to  our  intuition.  Unfortunately,  this  is  the 
exception. 

It  can  be  shown  that  the  locus  of  all  points  where  the  MAV’s  course  is  or¬ 
thogonal  to  the  segment  \BC\  is  a  circle  whose  diameter  is  the  segment  | BC\.  From 
points  outside  this  circle  and  between  the  normals  to  \BC\  erected  at  B  and  (7,  the 
MAV  trajectories  given  by  the  solution  of  the  differential  Equation  (18.181)  initially 
migrate  toward  the  orthogonal  bisector  \BC\.  However,  once  inside  the  said  circle, 
the  trajectories  flow  toward  B  or  (7,  and  a  collision  with  one  of  the  obstacles  occurs. 
Also,  trajectories  initially  to  the  left  of  the  orthogonal  bisector  to  \BC\  erected  at  (7, 
terminate  at  (7.  Trajectories  initially  to  the  right  of  the  bisector  to  \BC\  erected  at 
B,  terminate  at  B.  Thus,  a  collision  with  an  obstacle  is  in  fact  unavoidable!  The 
annotated  numerical  solution  is  illustrated  in  Figure  18221 

3.3  Analysis  2  -  Omni  Directional  Camera 

Figure  18.81  shows  the  geometry  for  the  optic  flow  measurement-based  obstacle 
avoidance  system  using  one  omni  directional  camera.  The  optical  flow  in  the  camera’s 
focal  plane  generated  by  obstacles  B  and  (7,  respectively,  is  measured.  Recall  that 
the  objective  is  to  steer  the  air  vehicle  between  the  obstacles.  The  camera’s  focal 
plane  is  shown  offset  from  the  y-axis  by  the  focal  distance  f,  and  the  projection  of 
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Trajectories  for  camera  FOV  of  360 


Figure  3.2:  Flow  field  when  equilibrating  LOS  rates.  The  trajectories  terminate  at 
obstacles  B  and  C. 

obstacles  B  and  C  in  the  camera’s  field  of  view  onto  the  image  planes  is  at  fB  and 
fc,  respectively  ra 

The  coordinates  in  the  camera’s  focal  plane  of  the  objects  in  the  images,  B  and 
C  are  fB  and  fc,  respectively.  The  optical  flows  generated  by  B  and  C  are  fB  and 
fc-  In  the  image  planes  the  obstacles  are  at 


fB  =  f  ttmf 
fc  =  -f  tan  (A  -  </>) 


and  the  optic  flow  rates  are  calculated  as 


Ib 

fc 


-f 


cos2(v4  —  0) 


(A  -  4>). 


(3.14) 

(3.15) 
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Figure  3.3:  Problem  geometry  for  one  omnidirectional  camera 

The  biomimetic  inspired  guidance  law  in  Equation  (13.161)  relies  on  nullifying  the  dif¬ 
ference  between  the  optical  flows  generated  by  B  and  C  (Equations  (13.141)  and  (??) 
respectively).  The  MAV’s  turning  rate,  0,  is  set  accordingly.  Thus,  the  turning  rate 
(j)  or  control  of  the  MAV  is  determined  by  the  guidance  law 

fB  =  ~fc  (3.16) 


Inserting  Equations  (13.141)  and  (13.141)  into  Equation  (13.161)  yields 


COS2  (j) 


cos2(t4  —  (p) 


(i-0) 


(3.17) 
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we  can  write 


sin  LA 
cos  LA 


d  .  a 

—  smfl 

R 

r  —  d  cos  6 
R 


where 


R  =  Vr2  +  d?  —  2 dr  cos  0 


(3.18) 


From  Equation  (18.171)  we  calculate 


cos2  0 


cos2  0  +  cos2  (A  —  0 ) 


A 


COS2  0 


( 1  +  cos2  A)  cos2  0  +  sin2  A  sin2  0  +  2  sin  A  cos  A  sin  <fi  cos  <fi 

1 


A 


1  +  cos2  A  +  sin2  A  tan2  0  +  2  sin  A  cos  A  tan  0 

1 


A 


1  +  (sin  A  tan  0  +  cos  A)2 


A. 


(3.19) 


Hence 


cosH  0 


d 


1  +  (sin  A  tan  0  +  cos  A)2  dt 

1 


'sin  A) 


1  +  (sin  A  tan  0  +  cos  A)2  dt K  R 


d  d  . 

(—  sint)) 


(3.20) 


and  therefore, 


r  —  d  cos  6 


d 


R  cos  66  —  sin  6R 


1  +  (sin  A  tan  0  +  cos  A)2 
d 


1  +  (sin  A  tan  0  +  cos  A)2 


R2 

R  COS  66  -  sin  grf-drcoeO+drsinOO 

- (3.21) 


R2 
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where  substituting  Equation  (13 .2 lft  into  Equation  (18.191)  gives 


d 


9{R2  cos  9  —  dr  sin2  0)  +  r  sin  9(d  cos  9  —  r) 


R2  1  +  (sin  A  tan  0  +  cos  A)2 


d 


1 


(0 


r  —  d  cos  9 
■r2  cos  9  +  d2  cos  9  —  dr  —  dr  cos2  9 


R2  1  +  (sin  A  tan  0  +  cos  A)2 
d  1 

R2  1  +  (sin  A  tan  0  +  cos  A)2 
Vd  f  sin0  +  sin(0  —  0) 

R?  1  +  (sin  A  tan  0  +  cos  A)2 
Vd  f  sin  0  +  sin(0  —  0) 

Id2  1  +  (Jsin0tan0+  ^f^)2 


r  —  d  cos  9 


[(r  cos  9  —  d)9  —  r  sin  9] 


Vd 

Vd 


-  sin0  +  sin(0  —  0) 


r2  +  d2  —  2 dr  cos  9  +  (d  sin  9  tan  0  +  r  —  d  cos  9 )s 
f  sin  0  +  sin(0  —  0) 


r2  +  d2  —  2dr  cos  9  +  [  dcos(^+^)  _|_  r]2 

L  COS  <j)  2 

V/d  sin0  +  (^)  sin(0  -  0) 

77?  1  +  (5)2-2(5)coS9  +  [^^  +  ^ 


r  sin  9) 


(3.22) 


Set 


r  V 

r  — »  t  — >  —t 


d’  d 

Hence,  the  non-dimensional  closed-loop  dynamics  are 


r 

9 

4> 


—  cos  0,  r(0)  =  r0 

— -sin0,  0(0)  =  0O 

r 

1  sin  0  +  r  sin(0  —  0) 

r  1  +  r2  —  2r  cos  0  +  [  ~  cos^~l~<^^  +  r]2 

L  COS  (j)  J 

cos2  0  sin  0  +  r  sin(0  —  0) 

r  (1  +  r2  —  2r  cos  0)  cos2  0  +  [—  cos(0  +  0)  +  r  cos  0]2  ’ 


(3.23) 

(3.24) 


(3.25) 


where  0(0)  =  0o,  0  <  t.  The  MAV’s  kinematics  are  governed  by  the  set  of  three 

nonlinear  differential  Equations  (13.231)  -  (13.251). 


It  is  easy  to  see  that  if  the  initial  configuration  is  symmetric,  as  in  Figure  13141 
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A 


1/2 


c 


where 


Figure  3.4:  Symmetric  initial  geometry 


cos  6*o  =  - —  (3.26) 

2ro 

sin  0o  =  77-  (3.27) 

zro 

then  the  configuration  stays  symmetric.  That  is, 


cos(6*(f))  = 

(3.28) 

or; 

sin  (<t>(t))  = 

2  r(i)-0£(' 

(3.29) 

Consider  the  following  question:  If  0  <  0o  <  arcsin(l/2ro),  does  the  trajectory 
avoid  B ? 
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To  answer  this  question  proceed  as  follows:  If  9  is  monotonically  decreasing,  one  can 
eliminate  t  and  obtain  a  set  of  two  differential  equations: 


dr 

d9 

dcf) 

de 

m) 


r 

tan  0’ 


r(0o)  =  r0 


_ 1  +  r(S~cos^) _ 

1  +  r2  —  2r  cos  8  +  (sin  9  tan  0  —  cos  9  —  r)2  ’ 
0o,  0  <  9  <  90 


(3.30) 


(3.31) 


Let 


then 


(3.32) 


Hence, 


c  = 


1 


tan2  0  cos2  0 


sin2  0 


(3.33) 


dr 

d9 

d( 

d9 

C(«o) 


r  C  .  r(8 0)  =  r0, 

,2.  1  +  r(Csin6)  —  cosd) 

cos2  0(1  +  r2  —  2r  cos  9  +  (^^  —  cos  9  —  r)2)  ’ 

— ,  0  <  8  <  0O. 
tan  0o 


(3.34) 


(3.35) 


The  flow  field  is  obtained  by  solving  the  differential  system  Equation  (13.341)  and  (13.351) 
and  plotting  x(t)  =  r(t )  cos(t),  y(t)  =  r(t )  sin  6(t),  0  <  t.  In  general,  the  flow  field  is 


given  by  the  solution  of  the  differential  system 


—  COS  0  , 

'o 

II 

o 

(3.36) 

—  sin  0  . 
r 

JO 

II 

o 

(3.37) 

COS2  0 

sin  0  +  r  sin(d  —  0) 

r  (1  +  r2  —  2r  cos  0)  cos2  0  +  [—  cos(d  +  0)  —  r  cos  0]2  ’ 


0(0)  =  0o,  0<t  (3.38) 


for  0  <  6*o  <  f ,  0  <  r0  <  2co1sgo  and  —  90  <  0O  <  7r  —  6>0,  and  for  |  <  90  <  n, 
0  <  r0  <  oo  and  —  60  <  0O  <  7r  —  60. 

The  “solution”  of  the  differential  Equations  (I3.36IMI3.38I)  shows  that  the  MAV 
flies  between  the  obstacles  and  the  maneuvers  are  not  too  extreme.  The  trajectories 
avoid  the  obstacles  for  all  initial  positions.  Obstacle  avoidance  is  then  guaranteed 
which  is  seen,  for  example,  in  Fig.  13.51  ,  where  the  MAV  is  initially  positioned  at 
various  x  coordinates  of  0  to  1  at  0.25  intervals  and  the  MAV’s  initial  course  angle  is 
do  +  4>o  =  f  •  That  is,  the  MAV  is  pointed  in  a  direction  orthogonal  to  the  segment 
\BC\.  The  MAV  will  approach  the  orthogonal  bisector  of  the  segment  |  BC\  which 
connects  the  obstacles,  as  shown  in  Fig.  13.51 

When  the  UAV  is  initially  positioned  on  the  orthogonal  bisector  of  the  segment 
\BC\,  the  trajectory  appears  to  be  stable.  Moreover,  the  “ideal”  symmetric  trajecto¬ 
ries,  namely,  the  trajectories  emanating  from  the  orthogonal  bisector  of  the  segment 
\BC\  and  with  initial  course  angle  0o  =  arcsin(l/2ro),  and  where  intuition  would 
predict  smooth  sailing  between  the  obstacles,  are  in  fact  unstable!  This  is  proved 
in  Section  13.4.21  in  the  sequel.  The  end  result  is  that  the  obstacles  are  avoided  but 
unstable  for  short  ranges  from  B  and  C. 

3.3.1  Looming  Theory.  Two  features,  say  B  and  C .  might  in  fact  represent 
the  edges  of  a  solid  object  and  the  segment  \BC\  the  surface  of  the  object.  In  this 
case,  there  is  one  obstacle  in  the  field  of  view  of  the  camera,  and  it  is  the  solid  segment 
\BC\.  To  address  this  situation,  the  concept  of  looming  becomes  complicated.  The 
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Optic  Flow  T rajectories  for  Camera  FOV  of  360 


Figure  3.5:  This  figure  depicts  Trajectories  Using  Optic  Flow  Balancing 

work  is  confined  to  a  planar  geometry  example.  The  obstacle  is  a  solid  white  object 
with  texture  that  is  defined  by  two  closely  located  dark  features  at  points  B  and  C. 
Thus,  the  segment  \BC\  does  not  represent  an  opening  but  rather  a  solid  line.  The 
distance  d  =  \BC\  <<  1.  As  before,  the  geometry  is  similar  to  that  of  the  optic  flow 
balancing  analysis  -  see  Fig  13.31  Once  again,  the  positions  of  the  features  B  and  C  in 
the  camera’s  focal  plane  are 


/b  =  f  tan  0, 
fc  =  -f  tan  (A  —  f>), 


and  the  optic  flow  rates  are  calculated  as 


/b 

fc 


cos2  (j)  ' 


-f 


1 


cos2  (A  —  <p) 


(A-  <t>) 


(3.39) 

(3.40) 
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Now,  track  fB  and  fc  features  over  time  and  describe  looming(L)  as 


L  = 


Jb  —  fc 

f 


(3.41) 


where  /  is  the  focal  length  of  the  camera  thus  looming  can  be  written  as, 


fR-  fr  1  1  • 


f  cos2  0  cos2  (A  —  0) 


(3.42) 


At  long  range  (if  |A|  <<  1)  then  looming  is  approximated  as, 


L 


COS2  0 


A. 


(3.43) 


It  is  useful  to  note  that 


1 


COS2  0 


1  +  (7b)2' 


(3.44) 


When  the  aircraft  bears  down  on  the  bisector  of  the  segment  \BC |,  the  ZA  is  in¬ 
creasing.  This  intuitively  means  that  A  >  0  =>  fB  —  fc  >  0.  Hence,  monitoring 
fB  ~  fc  >  0  indicates  that  segment  | BC\  is  indeed  closer.  To  determine  L,  calculate 


,  d  .  . 
A  =  —  sm  6 
R 


and  taking  the  derivative  yields 


A  =  d 


R  cos  00  —  R  sin  0 

A2 


(3.45) 


(3.46) 
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where 


R  =  -Vcos(<j>- A) 


(3.47) 


9  = 


r 


(3.48) 


So  A  becomes, 


A 


^ T-  [—  cos  9  sin  0  +  sin  9  cos(0  —  A)] 
R2  r 


(3.49) 


and, 


R  sin# 
r  sin(6l  +  A) 


Then  substituting  Equation  (13.501)  into  A  and  simplifying  gives, 


(3.50) 


A  = 


dV 


R?  sin(6*  +  A) 
dV  sin  9 
R?  sin(d  +  A) 
dV 


[—  sin  9  cos  9  sin  0  +  sin  9  sin(#  +  A)  cos (0  —  T)] 
[cos  9  sin  0  +  (sin  9  +  A  cos  9)  (cos  9  +  A  sin  0)] 


=  — -  (1  —  A  cot  9)  [—  cos  9  sin  0  +  sin  9  cos  0  +  A  cos (9  —  0)] 
R 2 

=  -5jf  (sin(0  “  4>)  +  7l[cos(0  -  0)  -  sin(d  -  0)]) 

R2  sm  9 


dV 

~r? 


.  ,n  ,,  .sin 
sm(d  —  0)  +  A— 


sin  9 


Thus, 


;  dV  .  (/.  ^ 

A  «  —  sm(6l  -  0) 
R2 


Hence,  substituting  A  into  Equation  13.431  yields 


(3.51) 


L  = 


Ib  ~  fc  dV  sin(0  - 


/ 


R?  cos2 


(3.52) 
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One  exception  is  when  0  =  0,  then 


T  ,  dV  .  n  d- 

L  =  A  =  _slae  =  --e 


(3.53) 


Another  interesting  aspect  to  inspect  is  that 


(3.54) 


In  this  case,  when  the  camera  moves  toward  the  features  on  the  orthogonal  bisector 
of  segment  | BC\  it  can  be  seen  that, 


(3.55) 


Hence,  by  monitoring  /b  +  fc  7^  0  it  can  be  determined  that  the  segment  1 50 1  is  not 
directly  in  the  line  of  sight.  This  is  key  when  considering  control  mechanisms  for  any 


objects  that  loom  in  the  FOV.  The  implementation  of  controls  in  the  case  of  looming 
occurs  maybe  dependent  upon  how  this  appears. 

3.3.2  Field  of  View  Restrictions.  FOV  limitations  are  included  such  that 
the  FOV  is  set  at  120°  at  ±60°  from  the  heading  in  each  direction  -  see  Fig.  13.61  This 
complicates  the  scenario  as  the  obstacles  may  or  may  not  always  be  within  the  FOV. 
When  both  obstacles  are  within  the  FOV,  the  previously  discussed  solution  applies 
the  0  control  to  equalize  LOS  to  each  obstacle.  Unfortunately,  the  ensuing  trajectories 
generate  heading  angles  which  invariably  cause  an  obstacle  to  be  outside  the  FOV. 
The  previously  derived  guidance  law  does  not  apply,  and  a  new  guidance  law  is  called 
for.  Therefore,  if  the  line  of  sight  does  not  rotate,  it  suggests  a  collision  is  imminent, 
so  an  increase  of  the  turning  rate  0  is  commanded,  specifically,  0  must  become  larger 
to  ensure  no  collision  takes  place.  The  turn  rate  is  relative  to  the  location  of  the 
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obstacle.  For  example,  if  the  obstacle  is  to  the  left  of  the  MAV  then 

0  =  0-^  (3-56) 

and  if  it  is  to  the  right  of  the  MAV  then 

0  =  0+^  (3.57) 

where 

*  =  To  <3'58) 

This  guidance  law  is  secondary  to  the  first  and  once  the  second  obstacle  is 
reacquired  in  the  FOV,  the  first  law  has  precedence.  Even  though  the  first  guidance 
law  has  preference,  the  results  of  the  simulation  show  that  both  obstacles  rarely  stay 
within  the  FOV  of  the  camera  for  any  length  of  time.  Therefore  the  second  guidance 
law  determines  the  MAV’s  trajectory.  Interestingly  enough,  this  always  results  in  an 
unimpeded  trajectory  -  see  Fig.  (13.71b 

An  additional  guidance  law  is  needed  to  handle  the  case  when  there  are  no 
obstacles  in  the  FOV.  This  law  returns  the  trajectory  to  its  original  heading  which  is 
orthogonal  to  the  line  between  the  obstacles.  This  is  intuitive  because  any  trajectory 
that  is  temporarily  modified  for  the  purpose  of  obstacle  avoidance  must  return  to  the 
original  planned  path. 

3. 3. 3  Many  Obstacles  and  Range  Limits.  The  next  step  entails  the  handling 
of  multiple  obstacles  or  features,  and  limiting  the  range  of  the  camera’s  FOV.  This  is 
accomplished  to  simulate  the  multiple  features  objects  in  a  realistic  scene  and  actual 
range  limitations  of  the  sensor.  The  features  are  in  the  left  and  right  hand  side  of  the 
FOV,  arranged  in  a  row,  with  one  directly  in  the  middle.  There  are  several  obstacles 
that  are  in  the  MAV’s  path  but  the  objects  must  be  within  the  range  limit  of  the  FOV. 
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Vehicle  with  120  degree  FOV 


-0.5 - 1 - 1 - 1 - 

-0.5  0  0.5  1  1.5 

X  Position 


Figure  3.6:  The  trajectory  is  shown  with  the  field  of  view  as  green  lines 


Trajectories  with  120  deg  FOV 


Figure  3.7:  The  trajectories  are  shown  with  the  field  of  view  restriction  of  120° 
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The  center  of  mass  of  the  feature  in  the  right  hand  side  of  the  FOV  is  calculated  and 
similarly,  the  center  of  mass  of  the  feature  in  the  left  hand  side  is  also  calculated. 
These  centers  of  mass  points  become  the  new  points  B  and  C  -  see  Fig.  13.31  These 
new  reference  points  are  then  subjected  to  the  previously  discussed  guidance  laws. 
This  is  a  dynamic  process  because  as  the  MAV  moves  through  the  scene,  the  average 
position  of  the  obstacles  will  change  -  see  Fig’s  13.91 13.101 13.111  and  13.81  The  average 
position  is  determined  by  both  the  position  of  the  vehicle  and  the  centered  obstacle. 
The  centered  obstacle’s  position  will  be  averaged  with  the  left  side  objects  if  it  is 
within  range  and  the  object  is  to  the  left  of  the  vehicle’s  heading.  Conversely,  it  will 
be  averaged  with  the  right  hand  side  objects  if  it  is  within  range  and  on  the  right 
hand  side  of  the  vehicle  heading.  This  produces  MAV  trajectories  that  avoid  all  the 
obstacles  in  the  scenario  -  see  Fig.  13.81 


Typical  MAV  Trajectories  in  a 
Multiple  Obstacle  Scenario  with  Camera  Range  Limits 


Figure  3.8:  The  trajectory  is  shown  with  center  of  mass  position  changes 
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MAV  in  a  Multiple  Obstacle  Scenario  with  Camera  Range  Limits 


Figure  3.9:  The  vehicle  here  is  shown  for  the  trajectory  initiating  from  point  (0.5,1). 
Typical  MAV  trajectories,  in  a  multiple  obstacle  scenario,  ±60°  FOV  and  sensor  range 
limit.  In  this  figure,  the  vehicle  can  be  seen  in  the  middle  of  the  obstacles  with  red 
asterisks  depicting  the  mean  location  for  all  the  obstacles  captured  from  the  camera 
and  green  lines  the  FOV. 


MAV  in  a  Multiple  Obstacle  Scenario  with  Camera  Range  Limits 


Figure  3.10:  The  vehicle  here  is  shown  for  the  trajectory  initiating  from  point 
(0.25,1).  In  this  figure,  the  vehicle  is  traversing  the  left  trajectory  and  the  obsta¬ 
cle  means  are  depicted  with  green  asterisks. 
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MAV  in  a  Multiple  Obstacle  Scenario  with  Camera  Range  Limits 


Figure  3.11:  The  vehicle  here  is  shown  for  the  trajectory  initiating  from  point 
(0.75,1).  In  this  figure,  the  vehicle  is  traversing  the  right  trajectory  and  the  mean 
obstacles  are  the  purple  asterisks.  These  asterisks  will  update  for  all  examples  as  the 
vehicle  moves  through  the  scene  and  new  obstacles  can  be  captured  while  others  will 
fall  out  of  the  FOV. 

3.4  Stability 

The  fact  that  the  “ideal”  trajectories  emanating  from  the  orthogonal  bisector  of 
the  segment  \BC\  and  with  initial  course  angle  b0  =  arcsin(l/2r0)  are  unstable  is  re¬ 
vealed  by  an  application  of  Lyapunov’s  first  method.  The  dynamics  matrix  is  obtained 
by  linearizing  the  dynamics  about  the  nominal  ideal  trajectory  and  its  eigenvalues  are 
shown  to  be  unstable. 

3-4-1  Line  of  Sight  Rate  Balancing  Guidance.  The  nonlinear  dynamics  are 
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0 


tan  cf) 


—  COS  (f), 
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r  sin  6* 

2  r2  +  1  —  3r  cos  6 
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To  investigate  the  stability  of  the  “ideal”  flight  path,  which  is  the  orthogonal  bisector 
of  the  segment  \BC |,  we  again  use  the  first  method  of  Lyapunov.  Linearization  yields 
the  dynamics  matrix 


sin0|J  sin0§| 

sin  4>  cos  cf>  dcf)  cos  cf>  dcf) 

r2  r  d6  r  86 


orthogonal 

bisector 


We  must  calculate  the  partial  derivatives 
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The  linearized  dynamics  matrix  which  is  parameterized  by  r,  is 


A(r)  = 


1  1-2  r2 
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The  characteristic  equation  for  the  determinant  of  A(r)  is  the  quadratic  equation 
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A  is  real 


49 


and 
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A  is  complex,  the  modulus  |A|  =  — 7= —  and  Re(A)  = -  ; 

7  2\/2r3  4r2v4r2  —  1 


The  modulus  of  the  complex  eigenvalues  is  plotted  in  Figure  EH21 
In  summary, 


Re(A)  >  0  V  r  > 


1 

2 


Both  eigenvalues  are  unstable.  The  eigenvalues  of  the  linearized  dynamics  matrix  A 
are  unstable  at  any  range. 


3-4-2  Optic  Flow  Balancing  Guidance.  The  nonlinear  dynamics  are  in  part 
specified  by  the  function  -  see  Equation  (18.381). 
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We  need  to  evaluate  the  partial  derivatives  of  /  along  the  orthogonal  bisector  of  the 
segment  \BC\,  where 
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Note  that  the  expression  in  the  denominator  of  the  function  /,  > 
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Hence,  we  calculate  the  partial  derivatives  along  the  orthogonal  bisector 
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Similarly, 
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and, 
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The  application  of  Lyapunov’s  first  method  requires  one  to  obtain  the  linearized 
dynamics  matrix  A.  The  linearized  dynamics  matrix,  which  is  parameterized  by  r 
and  is  (r  >  |),  is 
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To  find  the  eigenvalues,  set 
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which  gives  the  characteristic  Equation 
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Setting  A  :=  -  yields  the  simpler  cubic  equation 


/(A)  =  A3  +  a\2  +  a  =  0, 


(3.59) 


where 


2 

a  =  .  =. 

\/4r2  —  1 

The  roots  of  of  the  cubic  polynomial  are  investigated.  First  observe  that  /(A)  — » 
— oo  for  A  — >  — oo  and  we  calculate  /(—a)  =  a  >  0.  Hence,  there  exists  a  real  root 

Ai  <  —a. 

Furthermore,  the  product  of  the  cubic  polynomial’s  three  roots  is  the  negative  of  the 
free  coefficient  of  the  cubic  polynomial,  namely,  it  is  —a.  From  this  we  conclude  that 
the  product  of  the  remaining  two  roots,  A2A3  >  0.  Hence,  the  remaining  two  roots 
are  either  a  complex  pair,  or,  if  the  roots  are  real,  they  are  both  positive  or  are  both 
negative.  Next,  the  sum  of  the  roots  of  the  polynomial  is  the  negative  of  the  coefficient 
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of  A2,  namely 


Ai  +  A2  +  A3  —  —a 


Hence 


A2  +  A3  —  —a  —  Ai 

>  —a  —  (— a ) 

=  0 

Therefore,  the  roots  A2  and  A3  of  the  polynomial  are  either  positive  or  are  complex 
with  a  positive  real  part.  Next,  differentiation  of  the  function  /(A)  and  setting  the 
derivative  to  zero  gives  the  quadratic  equation  in  A:  ,  3A2  +  2a\  =  0.  A  second 
differentiation  reveals  that  the  root  A  =  —  |a  is  a  local  maximum  and  the  root  A  =  0 
is  a  local  minimum.  Since  /( 0)  =  a  >  0,  we  deduce  that  the  roots  A2  and  A3  are  a 
pair  of  complex  roots  with  a  positive  real  part  Vr  >  |. 

Hence,  similar  to  the  LOS  rate  balancing  guidance,  also  optic  flow  balancing 
guidance  causes  instability  at  any  range.  As  shown  in  Figure  13.121  the  guidance 
instability  becomes  worse  at  closer  ranges. 

3. 5  3-D  Simulation 

This  section  applies  the  theories  and  analyzes  how  well  it  performs  in  a  slightly 
more  complicated  3-D  case.  The  3-D  case  is  broken  down  into  a  two  channel  approach. 
One  channel  being  horizontal  and  the  other  being  vertical.  All  of  the  commands 
are  predicated  on  the  assumption  that  the  center  of  the  FOV  is  the  focal  point  for 
all  possible  collisions  and  the  path  in  which  the  vehicle  is  headed.  Then  the  focal 
plane  is  halved  vertically  for  the  horizontal  channel  and  horizontally  for  the  vertical 
channel.  Each  channel  applies  the  LOS  equalization  and  looming  detection  algorithms 
to  control  the  vehicle  trajectory  through  the  scene.  The  scene  is  generated  using 
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Stability  Plot  for  OF  Control  Equation 


Figure  3.12:  The  figure  displays  the  roots  to  the  stability  Equation  (13.591)  given  a 
value  of  r 

MATLAB®.  An  environment  is  created  with  buildings  and  other  possible  obstacles, 
which  are  created  in  3-D  with  multiple  points  established  for  calculation  purposes. 
Within  the  scene,  an  air  vehicle  is  depicted  as  a  point  in  the  3-D  space  and  is  iterated 
through  the  scene  by  adding  translation  velocity  values  at  each  time  step.  The  scenario 
is  then  iterated  a  number  of  times  to  complete  the  path  through  the  entire  scene  and 
return  the  vehicle  to  the  preplanned  trajectory.  The  calculation  of  optic  flow  is  then 
accomplished  and  stored.  The  stored  information  is  then  exploited  using  segmentation 
and  image  morphology.  All  parameters  are  derived  for  LOS  rate  equalization  and 
looming.  The  control  law  then  uses  the  information  given  to  update  the  control 
parameter  0.  Then  the  position  is  updated  and  repeated  through  the  scene  -  see 
Fig.  13.131  From  this  simulation,  it  will  show  that  the  LOS  equalization  and  looming 
detection  with  established  control  law  provides  for  a  possible  solution  for  collision 
avoidance. 
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Figure  3.13:  Flow  chart  depicting  the  process  for  which  the  theory  is  implemented 
in  a  3-D  scenario. 

3.5.1  Simulation  Environment.  The  simulation  is  conducted  in  MATLAB®. 
The  MATLAB®  environment  easily  handles  large  matrices.  This  makes  it  perfect  for 
the  simulation  due  to  the  scene  information,  iterations  of  position  update,  and  itera¬ 
tions  of  the  optic  flow  which  all  require  large  matrix  manipulation.  The  simulation  is 
a  series  of  iterations  that  update  vehicle  and  camera  positions  in  the  scene,  calculates 
ranges  from  the  focal  plane  to  the  obstacles  in  the  scene,  compares  current  ranges  to 
the  previous  iterations  of  each  range  to  calculate  a  translation  in  3-D  space  for  the 
calculation  of  the  optic  flow  fields.  The  simulation  captures  what  would  be  seen  from 
real  imagery  with  an  application  of  a  good  optic  flow  algorithm.  The  following  will 
describe  the  scene,  vehicle  trajectory,  and  the  optic  flow  development  in  more  detail. 
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3.5. 1.1  Scene  Generation.  The  scene  itself  is  a  series  of  polyhedra 
that  are  arranged  in  a  Cartesian  coordinate  plane.  These  polyhedra  or  obstacles,  are 
placed  in  various  locations  and  at  varying  heights,  widths,  and  depths.  The  scene  is 
meant  to  resemble  an  urban  environment  in  which  there  are  many  buildings  and  where 
a  MAV  would  typically  be  used  -see  Fig.  13.141  This  urban  scene  can  be  modified  to 
meet  multiple  different  looks,  which  was  ideal  for  performing  numerous  runs  through 
the  scene  for  testing  the  algorithms  and  control  law  (see  Chapter  ITVl) . 


3-D  Scene 


20  0  Y  Position 

X  Position 


Figure  3.14:  Generic  Urban  Scene 


The  scene  generation  is  a  fairly  straight  forward  process.  It  begins  with  devel¬ 
oping  the  XY  plane  for  the  horizontal  channel.  The  XY  plane  was  from  0  to  20 
dimensionless  units,  creating  a  square  grid  with  a  matrix  that  incremented  in  steps  of 
0.1.  Next,  the  midpoint  was  found  in  this  plane  for  use  in  positioning  each  obstacle. 
To  position  an  obstacle,  the  Z  coordinate  must  be  defined  in  the  context  of  2-D  space, 
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therefore  a  midpoint  was  found  using  the  length  of  the  XY  plane  and  dividing  it  by 
two.  Subsequently,  a  Z  matrix  of  same  size  as  X  and  Y  was  constructed.  Now  that 
there  was  a  midpoint  and  a  Z  matrix,  the  values  in  this  matrix  were  set  by  selecting 
a  range  of  X  and  Y  matrix  positions  and  setting  the  Z  matrix  in  these  locations  to  a 
specific  value.  Once  the  entire  Z  matrix  values  are  set,  the  X,  Y,  and  Z  matrices  are 
used  to  extrude  obstacles  into  3-D  space.  These  steps  are  repeated  for  the  vertical 
channel  except  that  it  began  with  a  grid  of  the  YZ  plane,  and  extruding  in  the  X 
plane. 


3. 5. 1.2  Vehicle  Trajectories  and  Position.  The  vehicles  position  and 
trajectories  are  calculated  from  an  initial  position  and  orientation.  The  position 
and  orientation  are  both  given  in  vectors  containing  three  values.  For  the  position 
the  three  values  correspond  to  the  3-D  rectangular  coordinate  system  x,  y,  and  z 
respectively.  The  orientation  vector’s  three  values  correspond  to  pitch,  roll,  and  yaw. 
The  orientation  is  needed  specifically  for  updating  the  orientation  of  the  camera. 
This  ensures  that  the  optic  flow  calculations  made  are  indeed  for  the  objects  directly 
in  front  of  the  vehicle  (see  Section  13.5.31) .  Then  two  motion  vectors  are  initialized 
that  describe  the  motion  of  the  vehicle.  The  first  vector  represents  translation  in  the 
positive  x,  y,  and  z  directions.  The  second  vector  is  the  pitch,  roll,  and  yaw  values 
respectively.  The  translation  vector  was  initially  set  with  a  velocity  that  portrays 
the  preplanned  navigation  path.  The  orientation  motion  vector  is  initially  set  to  zero 
which  symbolizes  no  rolling,  pitching  or  yawing.  This  is  done  to  simplify  the  optic  flow 
calculations.  These  motion  vectors  are  used  to  update  the  position  and  orientation 
with  iterations  of  the  motions  added  to  both  position  and  orientation.  The  simulation 
increments  time  steps  by  updating  the  motion  vectors  in  each  iteration.  The  motion 
vectors  get  modified  based  on  the  control  law  which  attempts  to  maneuver  the  air 
vehicle  without  a  collision  (see  Section  13.5.41).  After  each  iteration,  the  position  is 
updated  and  the  new  position  is  plotted  in  the  scene  to  display  the  collision  free 
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trajectory  -  see  Fig  13.151  The  simulation  is  iterated  until  the  MAV  is  completely 
navigated  through  the  scene. 


MAV  T  rajectory 


Y  Position  -10-10  x  Position 


Figure  3.15:  Urban  Scene  with  Trajectory  Through  the  Scene 

3.5.2  Optic  Flow.  The  development  of  the  optic  flow  field  is  critical  to  being 
able  to  apply  the  LOS  equalization  and  looming  detection  algorithms.  To  calculate  the 
optic  flow  there  are  a  number  of  parameters  required  that  include  the  following;  focal 
plane  resolution,  FOV,  the  position  of  the  camera  relative  to  the  vehicle,  orientation 
of  the  camera  (roll,  pitch,  yaw),  and  the  range  of  the  camera’s  focal  plane.  The  focal 
plane  resolution  is  analogous  to  an  image  resolution  with  a  number  of  pixels  in  a  2-D 
plane.  It  is  initialized  by  a  2-D  matrix  with  even  number  of  rows  and  columns.  The 
FOV  is  established  as  120°  with  center  of  the  FOV  at  0°  and  with  ±60°  in  azimuth 
and  elevation.  This  is  done  in  vector  format  with  azimuth  first,  and  then  elevation. 
The  camera’s  position  is  calculated  in  3-D  rectangular  coordinates  and  the  camera’s 
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orientation  as  a  vector  of  the  three  components  roll,  pitch,  and  yaw,  identical  to 
the  vehicle’s  position  and  orientation.  The  camera’s  position  and  orientation  is  done 
relative  to  the  vehicle’s  center.  In  this  case,  the  camera  is  bore  sighted  with  the 
vehicle  and  all  of  the  values  are  set  to  zero  and  updated  identically  with  the  vehicles 
position  and  orientation.  The  range  is  set  to  one  for  all  pixels  in  the  focal  plane  so 
that  each  range  converted  easily  into  unit  vectors.  Now  that  all  of  the  parameters  are 
initialized,  some  of  these  will  be  updated  or  used  to  calculate  the  optic  flow. 

3.5.2. 1  Vectorization  of  Pixels.  In  order  to  determine  what  each  pixel 
captures  in  the  simulation,  the  FOV  is  divided  by  the  number  of  pixels  in  each  direc¬ 
tion  x  and  y  from  the  resolution  separately.  The  two  angle  values  then  become  the 
interval  angles  between  each  pixel  and  from  center  of  the  image.  This  is  accomplished 
where  the  x  interval  angles  are  azimuth  and  the  y  interval  angles  are  elevation.  The 
pixel  resolution  rows  correspond  to  the  elevation  (EL)  and  the  columns  correspond 
to  the  azimuth  (AZ).  These  are  then  used  to  calculate  the  unit  vectors  for  each  pixel 
in  the  following  way, 


Xpixel  COs(F/L/)sZ?7/(  AZ') 

Ypixei  =  cos  (EL)  cos  (AZ) 
Z pixel  sin(EX) 


(3.60) 

(3.61) 

(3.62) 

(3.63) 

(3.64) 

(3.65) 


Where  i  and  j  are  the  row  and  column  locations  in  the  matrix  representing  the  pixel 
location.  The  unit  vectors  D,j  are  known,  and  the  resolution  of  the  scene  must  be 
determined.  This  is  determined  by  using  the  length  of  the  scene  (20)  divided  by  the 
total  number  of  indexes  used  to  build  the  matrix  grid  for  each  X,  Y,  and  Z.  These 
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resolutions  are  set  into  a  vector  ( Xres ,  Yres ,  Zres)  and  multiplied  by  the  pixel  unit 
vectors  D;j  to  attain  a  new  set  of  pixel  resolution  vectors  (Rvx-  Rvy,  Rvz )  that 
are  a  fraction  of  the  unit  vectors  length.  Now  these  new  vectors  have  a  new  range 
vector  calculated  by,  Resmag  =  R‘yx  +  Rfy  +  Rfz  The  value  Resmag  is  explicitly 
how  the  range  for  each  pixel  of  one  can  represent  the  total  distance  in  the  scene. 
These  parameters  will  be  instrumental  in  calculations  for  the  next  section. 

3. 5. 2. 2  Range  to  Obstacles.  The  magnitude  of  the  range  vector  Resmag 
is  used  by  adding  this  value  along  the  pixel’s  range  vector  direction.  While  the  position 
of  the  camera  is  known  in  relation  to  the  scene  this  length  along  the  pixel  range  vector 
can  also  be  determined  relative  to  the  scene.  This  range  vector  is  added  in  succession 
until  the  relative  length  of  the  scene  is  reached  or  until  a  value  in  that  position  grid  of 
the  scene  is  reached.  This  represents  the  distance  to  this  obstacle  along  this  particular 
pixel  direction.  This  length  is  stored  as  a  range  value  for  that  particular  pixel  location. 
The  calculation  is  reaccomplished  for  each  pixel  in  the  FOV.  A  new  matrix  (M)  is 
calculated  by  dividing  one  by  the  range  to  get  a  matrix  the  same  size  as  range.  Then 
calculations  must  be  made  to  accommodate  the  translation  and  rotation  motions  of 
the  vehicle  and  update  the  pixel  unit  vectors.  This  range  is  then  used  to  create  a 
projection  into  a  unit  sphere  via  Equation. (13.661).  This  was  done  as  follows 


FF  = 

-M[T  -  (T  •  D)D]  -RxD 

(3.66) 

Xfiow 

FF  •  Fx 

(3.67) 

Yfiow 

FF  •  Fy 

(3.68) 

where  R  is  the  rotational  motion  (roll,  pitch  and  yaw),  D  is  the  pixel  unit  vector 
matrices  described  in  Equation.  (13.601).  T  is  the  translation  matrix,  FF  is  the  spherical 
projection  of  an  object  in  the  camera’s  view  [22],  Xfiow  is  the  magnitude  of  the  optic 
flow  in  the  horizontal  plane  for  each  pixel,  Yfiow  is  the  optic  flow  in  the  vertical  plane 
also  for  each  pixel,  Fx  is  the  3-D  matrices  for  the  unit  vector  out  the  right  of  the 
focal  plane,  Fy  similarly  is  matrices  for  the  unit  vector  out  of  the  top  of  the  focal 
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plane,  and  •  is  the  dot  operator.  With  these  two  values,  one  single  vector (Ofiow)  with 
magnitude  and  direction  is  calculated  by  combining  Xfiow  and  Yfiow  in  the  2-D  plane 
by  a  Euclidean  distance  calculation  Ofiow  =  \Jx‘jlow  +  Yfiow-  These  vectors  are  then 
plotted  and  updated  for  each  iteration  of  the  vehicle  motion  per  time  step. 

3.5.3  Obstacles.  One  of  the  most  important  steps  before  applying  the  con¬ 
trol  law  is  to  process  the  optic  flow  vectors.  This  is  achieved  using  image  processing 
techniques.  First,  the  flow  vectors  that  are  induced  by  objects  of  interest  must  be 
segmented  or  separated  from  the  rest  of  the  vectors.  Once  a  region  of  interest  (ROI) 
is  segmented,  image  morphological  techniques  are  used  to  combine  vectors  close  to¬ 
gether.  This  guarantees  that  objects  close  together  or  missed  vectors  in  between 
vectors  of  the  same  obstacle  would  be  evaluated  as  a  single  obstacle.  Then  a  centroid 
is  found  for  each  ROI  and  LOS  angles  and  looming  values  are  calculated  for  later  use 
by  the  control  laws. 

3.5.3. 1  Segmentation.  The  segmentation  of  the  flow  vector  field  is 
significant  for  the  correct  identification  of  obstacles  in  the  field  of  view.  This  stage  is 
critical  for  determining  how  many  ROIs  there  are  and  where  they  are  located  in  the 
FOV.  Segmentation  is  accomplished  in  three  different  ways,  one  by  applying  a  curl 
operator  to  the  vector  field  and  the  two  others  by  a  vector  averaging  technique.  This 
is  done  to  determine  which  would  provide  a  better  solution.  This  image  processing 
step  is  the  critical  point  in  the  development  of  the  control  law. 

The  first  technique  applied  is  the  calculation  of  the  curl  operator.  The  curl 
operator  is  a  gradient-based  expression  which  uses  magnitude  and  direction  changes 
between  flow  vectors  to  determine  possible  rotation  points  -  see  Fig.  12.81  This  method 
is  used  because  it  is  good  at  determining  changes  in  the  vectors  across  the  flow  field. 
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This  calculation  is  accomplished  in  the  following  way  based  on  Equation  02.371)  [21]  , 


v  ^  d(Xflow) 

xgradij  =  ^ 

(3.69) 

Yqrad  - 
r  graah]  — 

(3.70) 

curlij  =  X  gradij  —  Y  gradij 

(3.71) 

The  curlij  is  the  same  size  as  the  pixel  resolution.  Now  this  matrix  is  stored  and  then 
a  binary  mask  is  set  to  identify  the  ROI’s.  This  is  achieved  by  setting  ones  for  values 
greater  than  a  threshold  value,  rc  and  zero  for  those  less  than  rc.  This  threshold 
value  is  empirically  set  to  0.1  as  this  curl  value  is  identified  from  the  data  as  possible 
obstacles.  This  threshold  is  dynamic  as  it  would  change  with  varying  velocities.  Then 
the  mask  is  passed  into  the  morphological  algorithm  to  merge  ROIs  from  the  matrix 
for  simplification  of  the  obstacles  and  combining  closely  located  vectors  (see  Section 

m  m. 

The  alternate  methods  use  a  flow  field  averaging  technique.  This  simply  finds 
the  mean,  (/r),  of  the  flow  field  vector  values  for  the  entire  field.  This  is  calculated  by 

lJ  =  77  T  T  Di,i  (3.72) 

i=i  j= i 

where  N  is  the  total  number  of  pixels,  Pr  is  the  total  number  of  pixel  rows,  and  Pc 
is  the  total  number  of  pixels  columns.  Once  the  mean  value  is  established,  there  are 
two  ways  to  segment  the  obstacle  vectors.  First,  vectors  that  are  greater  than  /i  +  20- 
are  found  and  segmented  out  with  the  binary  mask.  The  quantity  a  is  the  standard 
deviation  and  is  calculated  by 

1 

2 

(3.73) 

Second,  any  vectors  that  are  larger  than  2 fi  are  segmented  out  as  mentioned  before. 
The  mask  of  the  flow  field  converts  the  index  locations  to  a  one  for  vectors  greater 


a  = 


N 


N  - 


2  'y  Fmean) 


2=1 
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than  2/jl  and  zero  for  all  others.  This  binary  mask  is  processed  with  a  morphological 
algorithm  to  merge  vectors.  This  is  accomplished  for  both  averaging  techniques. 


3. 5. 3. 2  Image  Morphology.  The  morphology  of  an  image  is  broken 
into  two  distinct  parts,  dilation  and  erosion.  “Dilation  adds  pixels  to  the  boundaries 
of  objects  in  an  image,  while  erosion  removes  pixels  on  object  boundaries”  [1]  -see 
Fig.  Era 
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Figure  3.16:  Illustration  of  Dilation  (reprinted  from  pQ) 


Erosion  removes  the  ones  on  the  boundary  of  the  matrix  shown  as  a  shaded 
region  in  Fig.  13.161  Each  operation,  dilation  and  erosion,  uses  a  structuring  element. 
The  structuring  element  shapes  the  dilation  and  erosion.  A  structuring  element  is  an 
arbitrarily  shaped  matrix  of  ones  or  zeros  p],  in  this  case  ones  are  used.  The  center 
pixel  of  the  element  will  set  the  pixel  location  of  the  mask  to  one  if  the  neighboring 
pixels  in  the  mask  are  ones.  In  this  case,  the  binary  mask  developed  in  Section 
13.5.3.11  is  the  binary  image  and  the  structuring  element  is  a  [1  x  3]  matrix  of  ones  - 
see  Fig  13.171  p].  This  dilation  is  then  applied  to  the  binary  mask.  Any  objects  close 
together  as  defined  by  the  structuring  element  are  morphed  into  a  single  object.  Then 
the  erosion  operation  is  applied  to  return  an  object  to  normal  size  maintaining  any 
object  now  joined  -see  Figs.  3, 18(a)[  3]18(b)[  and  3. 18(c 


Now  that  closely  located  objects  are  morphed  into  a  single  object,  each  of  the 
objects  are  labeled.  This  is  accomplished  by  a  function  in  MATLAB®  called  “BWLa- 
bel”  [Tj .  This  function  assigns  a  unique  number  to  each  ROI.  Now  that  the  ROIs  are 
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Figure  3.17:  Dilation  with  a  Structured  Element  (reprinted  from  jT] ) 


(a)  Binary  image  before  morphology 


Dilated  Binary  Image 


(b)  Dilated  image  (c)  Eroded  image 

Figure  3.18:  (a)  is  the  original  binary  image,  (b)  shows  the  binary  image  that  has 

been  dilated  joining  closely  located  objects,  (c)  displays  the  new  image  after  the 
erosion  process.  Notice  that  the  yellow  rectangles  highlight  areas  that  are  now  joined. 


64 


labeled,  the  centroid  for  the  region  is  found  using  “regionprops”  [lj.  This  finds  the 
centroid  of  each  ROI.  Non-integer  centroid  values  are  rounded  to  the  closest  integer 
value.  The  centroid  row  and  column  values  are  then  stored  for  use  in  calculations  in 
the  LOS  equalization  and  looming  detection  algorithm. 

3. 5. 3. 3  LOS  Equalization  and  Looming.  The  LOS  equalization  and 
looming  detection  algorithm  is  similar  to  that  in  Section  13.31  The  same  principle  is 
used  applying  the  guidance  control  law  using  <fi  to  equalize  the  LOS  changes  to  each 
object.  One  slight  difference  is  that  in  the  3-D  simulation,  the  vehicle  is  controlled 
by  an  input  of  translation  and  roll  with  a  simulated  focal  plane  being  derived.  This 
implies  that  9,  ZA,  and  A  can  be  calculated  directly.  So,  in  this  case,  Equation  (13.1711 
was  algebraically  manipulated  to  get, 

^  =  ,  cos  {A^W  ’  (3'74) 

COS  ((f))2 

where’  A  =  Ak-Ak_l.  (3.75) 


Before  this  could  be  evaluated,  the  parameters  9,  ZT,  and  A.  must  be  calculated.  To 
implement  3-D  collision  avoidance,  the  2-D  LOS  rate  equalization  equation  is  used  in 
two  channels,  horizontal  and  the  vertical  channels.  The  horizontal  channel  develops 
its  own  optic  flow  field  based  on  objects  in  the  X  and  Y  mesh  grid.  This  flow  field 
is  then  processed  as  mentioned  in  the  previous  sections.  Then  the  locations  of  the 
centroids  captured  are  used  to  seek  the  two  coordinates  closest  to  the  center  of  the 
FOV.  When  these  center  coordinates  are  found,  the  column  value  is  used  to  calculate 
the  two  angles  9  and  ^  (see  Section  13.31) .  The  index  value  for  column  of  the  centroid 
that  is  larger  than  the  center  indexed  value  corresponds  to  the  right  side  of  the  focal 
plane  and  is  used  to  find  9.  This  centroid  coordinate  is  then  used  to  pick  out  the  unit 
vectors  from  matrix  D  (see  Section  13.5.2.11)  and  calculate  9  as 


9  =  arctan  (  *’J’2  ) . 


(3.76) 
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The  index  value  for  the  column  of  the  centroid  that  is  smaller  than  the  center  index 
becomes  the  left  side  of  the  focal  plane  and  is  used  for  finding  ip  as 


ip  =  arctan 


/Aj,  2\ 


(3.77) 


Based  on  Figure  l3~3l  the  /PA  is  the  angle  from  A  to  points  B  and  C.  This  angle  was 
easily  calculated  as 


/PA  =  7T  —  (6  +  ip). 

The  final  parameter  left  is  <p  and  it  is  calculated  as 


(3.78) 


<j>  =  H-e,  (3.79) 

where  the  heading  H  is  the  actual  direction  of  movement  of  the  vehicle.  It  is  always 
initiated  at  H0  =  7t/2  and  updated  each  iteration  by  cp.  Now  that  all  of  the  param¬ 
eters  are  calculated,  the  LOS  rate  equalization  algorithm  has  everything  it  needs  to 
calculate  (p  for  the  control  law.  The  vertical  channel  is  implemented  almost  identically 
except  the  centroid,  is  substituted  for  the  row  instead  of  column.  Furthermore,  the 
bottom  half  of  the  focal  plane  is  analogous  to  the  right  side  plane  in  the  horizontal 
channel  and  top  half  same  as  left  side.  The  rest  of  the  calculations  are  identical  but, 
inserting  pitch  (Pq  =  0)  instead  of  heading. 

Next,  the  looming  algorithm  is  derived  in  the  same  manner  as  in  Section  13.3. li 
First,  the  segmentation  part  must  identify  a  centroid  in  the  center  of  the  FOV.  Then 
this  centroid  has  an  associated  minimum  and  maximum  value  corresponding  to  the 
edges  of  the  object  in  the  scene.  These  values  for  the  minimum  and  maximum  be¬ 
come  fc  and  fb  respectively  used  in  Equation  03.411) .  They  are  tracked  over  successive 
iterations  and  fc  and  fb  are  calculated.  The  looming  value  is  computed  from  Equa¬ 
tion  03.421)  for  the  current  iteration  and  stored.  The  looming  values  are  iterated  and  if 
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the  updated  values  are  larger  than  the  previously  stored  values  looming  is  occurring. 
Once  looming  is  detected,  the  control  law  is  used  to  make  a  direction  change. 

3.5.4  Control  Law.  The  Control  Law  is  the  compilation  of  all  the  infor¬ 
mation  devised  in  the  previously  stated  algorithms.  This  control  makes  the  actual 
changes  to  cj)  to  ensure  that  a  collision  is  avoided.  This  is  attained  in  a  step  by  step 
manner.  First,  decide  whether  there  is  an  actual  object  in  the  FOV  as  described  ear¬ 
lier.  If  there  is  an  object,  it  makes  a  choice  for  direction  change  depending  on  whether 
there  are  no  objects,  a  single  object,  or  two  objects.  This  correction  in  direction  is 
the  final  step  in  the  collision  avoidance  algorithm.  Each  decision  made  for  the  three 
possible  outcomes  can  be  seen  in  the  following  sections  and  Figure  I3T91 


Figure  3.19: 


Flow  Chart  Depicting  Control  Law  Process 
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3.5.4-  1  No  Detected  Obstacles.  The  case  where  no  object  is  detected 
requires  a  simple  decision.  For  this  case,  if  no  actual  object  is  detected  through  image 
processing,  the  decision  is  to  continue  on  the  pre-planned  path.  This  is  simple  if 
the  MAV  is  currently  in  the  direction  of  the  initial  conditons  ( H0/P0 )  then  0  =  0. 
Otherwise,  the  heading  or  pitch  0  must  be  changed  to  bring  the  direction  back  to 
H0/P0  and  is  dependent  on  the  current  heading  or  pitch.  For  heading  or  pitch  greater 
than  Hq/Pq ,  0  =  — 7t/18.  For  heading  or  pitch  less  than  H0/P0 ,  0  =  ir/18.  The 
changes  in  0  bring  the  vehicle  back  to  H0/P0  in  an  incremental  fashion,  where  A 
creates  a  quick  return  to  the  initial  direction.  This  is  because  no  obstacles  are  in  the 
FOV  so  it  brings  the  vehicle  back  to  the  initial  heading  and  pitch. 

3. 5. 4- 2  Single  Obstacle.  With  one  object  detected  in  the  FOV,  the 
control  law  needed  is  based  on  where  the  object  appears  in  the  camera  focal  plane. 
The  most  important  aspect  to  consider  is  the  instance  when  the  object  is  centered  in 
the  FO V(Cent).  This  implies  the  object  is  directly  in  front  of  the  vehicle  and  that  a 
collision  is  imminent.  If  looming  is  detected,  the  centroid  value  for  the  appropriate 
channel  (horizontal  or  vertcial)  is  used  to  determine  which  half  of  the  FOV  contains 
most  of  the  object.  Knowing  this  initiates  a  turn  in  the  opposite  direction  from  the 
object  -  see  Equationfl3.80D. 

if  centroid  >  Cent ,  0  =  7t/30,  (3.80) 

if  centroid  <  Cent ,  0  =  — 7r / 30,  (3.81) 

The  turn  rate  factor  A  js  representative  of  a  miniature  vehicle  turn  radius  to  make 
the  simulation  more  realistic.  For  the  case  when  the  object  is  clearly  in  the  left /top 
or  right/bottom  half  of  the  focal  plane,  the  commanded  control  is  to  turn  away  from 
the  object  in  the  following  way, 

0  =  7r/30  if  right/bottom  (3.82) 

0  =  — 7r/30  if  left/top  (3.83) 


68 


The  single  objects  commanded  control  is  essentially  based  on  which  half  of  the  plane 
the  object  fell  into,  but  if  the  object  is  centered  in  the  FOV  it  makes  the  same 
commanded  control.  The  singular  case  is  exaggerated  slightly  to  create  a  strong 
response  to  the  objects  in  the  scene.  The  control  for  multiple  objects  in  the  FOV  is 
more  complicated. 

3. 5. 4-3  Multiple  Obstacles.  The  scenario  with  two  different  objects  in 
the  scene  presents  a  unique  problem.  If  there  are  two  objects  in  the  same  half  of  the 
focal  plane,  or  in  opposite  halves,  how  should  the  MAV/UAV  be  commanded?  The 
problem  is  simplified  by  eliminating  extra  objects  detected  in  the  half  that  are  the 
furthest  from  the  center.  If  there  is  only  a  single  object  remaining,  it  is  controlled 
as  discussed  in  Section  13.5.4.21  If  there  are  two  objects  remaining,  but  in  opposite 
halves  of  the  focal  plane,  then  LOS  rate  equalization  is  called  for  and  0  is  adjusted 
according  to  Equation  (13.741). 

Now  that  all  the  possibilities  of  the  objects  in  the  scene  could  be  accounted  for 
and  the  commanded  control  is  implemented  in  the  previously  detailed  method,  the 
next  step  is  to  simulate  the  guidance  laws  developed  by  actual  implementation  in  a 
scene  that  mimics  a  real  environment.  The  next  chapter  will  describe  the  scenarios 
and  the  results  from  these  simulations. 
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IV.  Design  of  Experiments  and  Results 

4  •  1  Introduction 

Simulation  is  required  to  visualize  the  resulting  complicated  kinematics,  validate 
the  guidance  law  equalizing  the  LOS  rate,  and  to  accomplish  the  looming  calculation. 
Both  of  these  are  measured  with  optic  flow  from  a  3-D  scene.  The  interpretation  of 
the  optic  flow  field  and  then  applying  the  LOS  rate  balancing  guidance  laws  need  to 
be  analyzed,  to  gather  understanding  of  the  capability  of  this  obstacle  collision  avoid¬ 
ance  guidance  algorithm.  Analysis  of  the  trajectories  generated  in  these  simulations  is 
imperative  to  demonstrate  the  obstacle  avoidance  guidance  control  law.  In  this  chap¬ 
ter,  the  simulations  are  detailed  for  effectiveness  and  experimentation  configurations 
are  discussed,  (give  unbiased  results  to  support  the  claim  that  collision  avoidance  can 
be  achieved  with  LOS  rate  equalization  and  looming  detection.) 

4-2  Experiments 

The  experiment  entails  a  series  of  3-D  scenes  and  an  unmanned  air  vehicle  that 
navigates  through  each  scene.  Fourteen  different  scenes  are  generated  to  exercise 
the  guidance  law.  The  air  vehicle  is  initiated  at  various  locations  before  traversing 
the  scene.  The  various  initial  locations  are  used  to  demonstrate  as  many  different 
decision  points  as  possible  for  the  air  vehicle’s  guidance  law  and  to  analyze  the  logic 
of  the  control  choices.  The  vehicle  is  tracked  as  it  traverses  the  scene  and  position 
plotted  in  the  scene  and  displayed.  This  provides  visual  confirmation  of  a  collision 
or,  a  lack  thereof.  Success  is  based  on  collision  avoidance  alone;  any  collisions  in 
the  simulation  constitutes  a  failure.  The  14  scenes,  initializations,  and  trajectory 
mappings  are  discussed  in  the  ensuing  subsections. 

4-2.1  Scene  Descriptions.  The  scenes  are  developed  as  described  in  Section 
13.5.1.11  Here,  the  exact  layout  of  each  scene  is  detailed  and  how  it  might  represent 
a  realistic  scene.  The  first  scene  is  known  as  the  “  lattice”  -  see  Fig.  14.11  It  is  a 
trio  of  obstacles  in  the  vertical  direction  that  extrudes  up  from  the  XY  plane  to  the 
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Figure  4.1:  Scene  1  “lattice” 

maximum  height  of  Z ,  and  another  trio  of  obstacles  in  the  horizontal  direction  that 
extends  from  the  YZ  plane  across  the  scene  to  the  maximum  length  of  X.  Each 
one  of  these  obstacles  is  a  1  x  1  x  20  polyhedron.  Even  though  they  are  in  separate 
channels  for  optic  flow  calculations,  they  are  plotted  on  the  same  figure  and  give  the 
appearance  of  distinct  obstacles  in  3-D.  The  first  one  looks  like  a  large  tic-tac-toe 
board  and  the  second  one  like  a  large  cross.  The  cross  is  situated  behind  the  tic- 
tac-toe  and  center  out  on  its  middle  square.  This  configuration  splits  the  first  series 
of  obstacles  in  half,  which  is  done  to  exercise  multiple  instances  of  the  guidance  law. 
Scene  2  is  known  as,  the  “bridge”  -  see  Fig.  14.21  The  bridge  is  four  1  x  1  x  20  polyhedra 
in  the  XY  plane  and  1  x  5  x  20  polyhedra  in  the  YZ  plane.  The  four  obstacles  in 
the  horizontal,  XY  plane,  are  configured  with  two  having  equal  spacing  in  the  X 
direction  and  two  having  equal  spacing  in  the  Y  direction.  The  single  obstacle  in 
the  vertical,  YZ  plane,  is  extended  across  the  four  horizontal  obstacles  connecting 
them  and  giving  the  appearance  of  a  bridge.  Scene  3  is  known  as  the  “tunnel”  -  see 
Fig.  14.31  The  tunnel  is  square  and  is  shaped  by  four  separate  objects.  There  are 
two  objects  in  the  horizontal  plane  and  two  objects  in  the  vertical  plane.  They  are 
organized  in  a  way  that  it  creates  a  5  x  5  x  10  square  tunnel.  This  obstacle  challenges 
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Figure  4.2:  Scene  2  “bridge” 
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Figure  4.3:  Scene  3  “tunnel” 

the  vehicle  to  simultaneously  center  itself  vertically  as  well  as  horizontally.  Scene  4 
is  known  as  the  “stairsteps”  -  see  Fig  14.41  The  stairsteps  is  a  scene  that  has  the 
same  trio  of  obstacles  in  the  XY  plane  as  in  the  Lattice  scene,  but  it  also  has  six 
separate  polyhedra  that  extend  across  the  scene  in  the  horizontal  direction.  Three  of 
these  are  located  quite  low  and  appear  as  an  ascending  staircase.  The  other  three  are 
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Figure  4.4:  Scene  4  “stairsteps” 

placed  at  a  high  position  and  appear  like  descending  stairsteps.  The  challenge  here 
is  for  the  vehicle  to  climb  and  drop  to  center  vertically  while  still  avoiding  the  center 
obstacles  in  the  horizontal  channel.  Scene  5  is  known  as  the  “horizontal  wall”  -  see 
Fig.  14.51  The  horizontal  wall  is  in  the  horizontal  plane  and  runs  perpendicular  to 
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Figure  4.5:  Scene  5  “horizontal  wall” 

the  vehicle’s  direction  of  motion.  The  wall  is  quite  large  and  extends  across  most  of 
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the  scene.  This  simple  obstacle  will  completely  cover  the  entire  FOV  of  the  vehicle’s 
camera  for  most  locations.  This  is  probably  the  most  difficult  obstacle  to  avoid  and 
challenges  the  algorithm  in  low  contrast  environments.  Scene  6  is  the  “vertical  wall” 
-  see  Fig.  14.61  The  vertical  wall  is  similar  to  the  horizontal  wall  except  it  is  extends 
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Figure  4.6:  Scene  6  “vertical  wall” 

across  the  scene  from  the  YZ  plane.  This  wall  again  is  large  enough  that  it  will 
completely  blanket  the  FOV  of  the  camera.  This  obstacle  ensures  that  the  vertical 
control  channel  works  similar  to  the  horizontal  control  channel.  The  two  channels 
should  give  similar  results.  Scene  7  is  the  “horizontal  walled  alley”  -  see  Fig.  14.71 
This  scene  consists  of  two  walls  parallel  to  the  direction  of  the  vehicle’s  motion.  The 
two  walls  are  narrow  but  long  and  run  most  of  the  length  of  the  V-axis.  They  reach 
a  height  of  20  units.  This  obstacle  is  specifically  used  to  challenge  the  guidance  law 
for  its  ability  to  center  itself  through  the  alley  and  avoid  the  walls.  This  scenario 
is  sometimes  problematic  where  optic  flow  balancing  for  collision  avoidance  guidance 
is  used.  Scene  8  is  the  “vertical  walled  alley”  -  see  Fig.  14.81  This  scene  again  is  a 
replica  of  the  horizontal  walled  alley,  but  in  the  Y Z  plane,  and  exercises  the  vertical 
control  channel.  Scene  9  is  the  “horizontal  walled  alley  with  bars”  -  see  Fig.  14.91 
This  scene  is  exactly  the  same  as  the  horizontal  walled  alley  but  with  added  bars  that 
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Figure  4.7:  Scene  7  “horizontal  walled  alley” 
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Figure  4.8:  Scene  8  “vertical  walled  alley” 

extend  from  the  Y Z  plane.  These  extended  bars  are  designed  to  add  complexity  and 
challenge  the  guidance  law  to  stay  centered  horizontally  through  pitch  angle  changes. 
The  idea  is  that  slight  perturbations  might  destabilize  the  guidance  law  (see  Section 
I3.4.2D.  Scene  10  is  the  “row  of  buildings  creating  an  alley”  -  see  Fig.  14.101  This  scene 
is  two  series  of  five  2  x  2  x  20  buildings  that  are  aligned  parallel  to  the  direction  of 
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Figure  4.9:  Scene  9  “horizontal  walled  alley  with  bars” 
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Figure  4.10:  Scene  10  “row  of  buildings  creating  an  alley” 

the  vehicles  motion,  as  before.  This  scene  has  two  rows  of  buildings  creating  an  alley, 
but  this  time  with  discontinuities.  In  this  scenario,  the  vehicle  is  expected  to  traverse 
the  center  of  the  alley.  The  discontinuities  test  its  ability  to  handle  the  fluctuations 
in  the  optic  flow.  Scene  11  is  the  “sparsely  positioned  buildings  creating  an  alley”  - 
see  Fig.  14.111  In  this  scene,  the  concept  is  similar  to  that  of  the  last  scene  except, 
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Figure  4.11:  Scene  11  “sparsely  positioned  buildings  creating  an  alley” 

there  are  fewer  buildings  and  they  are  spaced  farther  apart.  The  buildings  are  still 
in  a  line  parallel  to  the  direction  of  vehicle  motion.  Scene  12  is  the  “row  of  narrow 
buildings  creating  an  alley”  -  see  Fig.  14.121  In  this  scene,  the  size  of  the  buildings  are 
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Figure  4.12:  Scene  12  “row  of  narrow  buildings  creating  an  alley” 

narrowed  down  to  1  x  1  x  20.  This  creates  less  optic  flow  vectors  that  can  be  used 
for  the  guidance  law.  This  shows  how  the  difference  in  detections  might  affect  the 
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guidance.  Scene  13  is  the  “up  down”  -  see  Fig.  14. 131  This  scene  consists  of  two  walls 
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Figure  4.13:  Scene  13  “up  down” 

from  the  vertical  plane.  The  first  wall,  closest  to  the  air  vehicle,  is  the  lowest  wall;  a 
20  x  2  x  10  polyhedron.  The  next  wall  farthest  away  is  the  same  dimensions  as  the 
first  wall  but  positioned  at  a  higher  elevation.  This  wall  again  is  10  units  high  but 
sets  10  units  above  the  surface.  The  title  of  this  scene  alludes  to  what  is  expected 
of  the  vehicle’s  trajectory.  It  tests  the  vehicle’s  ability  to  redirect  itself  and  handle 
large  obstacles  that  are  close  compared  to  those  same  obstacles  farther  away.  Scene 
14  is  the  “urban  scene”  -  see  Fig.  14.141  This  scene  is  considered  most  important 
and  is  designed  to  represent  an  urban  environment.  It  has  a  complex  arrangement  of 
buildings,  towers,  power  lines,  and  a  low  hanging  walkway.  The  buildings  and  towers 
are  in  the  horizontal  plane  where  power  lines  and  the  walkway  are  in  the  vertical 
plane.  The  buildings  are  constructed  in  the  manner  described  previously,  and  are 
offset  in  locations  to  make  it  difficult  to  traverse  the  scene.  The  buildings  also  vary 
in  height  and  width.  Towers  are  extended  from  the  top  of  two  buildings  and  one  is 
located  toward  the  rear  of  the  scene.  Each  tower  is  2  x  2  x  20  in  size.  The  power  lines 
drawn  using  the  smallest  increment  of  size  possible,  which  is  0.1,  and  in  the  Y  and 
Z  directions,  but  are  extended  across  the  X  axis  at  varying  lengths.  The  walkway  is 


78 


3-D  Scene 


20  0  Y  Position 

X  Position 


Figure  4.14:  Scene  14  “urban  scene” 

2  units  in  the  Y  direction  and  1  unit  in  the  Z  direction,  but  extended  across  three 
buildings  and  appearing  to  connect  them.  This  scene  is  the  most  complicated  and 
strives  to  challenge  the  vehicle’s  ability  to  avoid  all  obstacles  types  oriented  in  an 
extremely  cluttered  environment. 

4-2.2  Simulation  Initialization.  The  initialization  of  the  simulation  is  a  very 
important  step.  There  are  multiple  parameters  that  can  be  set  and  the  parameters 
include:  pixel  resolution,  velocity,  heading,  pitch,  position,  orientation,  and  scene. 
All  of  these  are  initialized  for  each  run  of  the  simulation.  Each  scene  is  created  and 
stored  previous  to  the  simulations  execution.  The  initialization  of  the  scene  is  the 
first  step  for  running  the  simulation.  The  stored  scene’s  information  is  loaded  just 
before  the  rest  of  the  parameters  are  set.  Then  the  initial  position  vector,  Pv,  is  fixed 
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for  the  run  by  randomly  choosing  the  Xq  and  Z0  location,  while  always  Y$  =  0. 


Pv  —  [X0  ho  Z0]  —  [m  0  n] 

Rv  =  [pitcho  rollo  yaw  o]  =  [0  0  0] 


(4.1) 

(4.2) 

(4.3) 


where  m  and  n  are  random  numbers,  generated  with  a  random  number  generator  on 


the  interval  [119]  to  keep  the  UAV  away  from  the  edges  of  the  scene  and  engaging 


obstacles.  The  X0  and  Z0  locations  are  randomly  initialized  to  create  an  unbiased 
experiment.  The  Yq  coordinate  remains  constant  at  0  because  the  vehicle  travels 
in  the  positive  Y  direction,  therefore,  initializing  at  this  position  is  necessary.  The 
orientation  vector  Rv  is  initialized  to  all  zeros.  This  is  indicative  of  wings  level,  heading 
due  north,  and  no  pitch.  This  starts  the  vehicle  in  a  nominal  position  starting  into 
the  scene  allowing  adjustments  from  there.  The  pitch  variable,  P,  is  set  to  0  while 
the  heading  variable,  H  is  set  to  |.  This  means  that  the  pitch  resets  for  level  flight 
in  the  XY  plane  and  the  heading  will  be  parallel  to  the  YZ  plane.  The  motion  of 
the  vehicle  is  commenced  without  rotation  motion,  Rin,  and  only  translation  motion, 
Ty,  in  the  Y  direction  with  a  magnitude  of  0.2.  The  initial  conditions  are  set  for  the 
translation  with  no  rolling,  pitching,  or  yawing  initiated  -  see  Equations  (14.41)  and 


Tin  =  [Tx  Tv  Tz]  =  [  0  .2  0] 

Rin  =  [pitch  roll  yaw ]  =  [0  0  0] 


(4.4) 

(4.5) 


Next,  pixel  resolution  is  set.  It  is  either  established  as  a  20  x  20  pixel  field  with 
resolution  of  400  pixels  -see  Figi4.151  or  as  a  50  x  50  pixel  field  with  a  resolution 
of  2500  pixels  -see  Fig.  14.161  The  second  option  is  referred  to  as  high  resolution, 
while  the  first  option  is  referred  to  as  low  resolution  in  this  thesis.  Finally  all  of  the 
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parameters  are  initialized  and  the  simulation  is  ready  to  run. 


Optic  FLow  Field 


Figure  4.15:  Low  Resolution  Optic  Flow  Field  of  the  Vehicle  Flying  Between  Two 
Obstacles  see  Fig.  13.151 

This  experiment  is  an  iterative  simulation  that  loops  100  times  to  complete 
trajectories  through  the  scene.  The  value  100  was  chosen  based  on  the  initialized 
translation  velocity.  If  Ty o  =  0.2,  then  lOO(Tyo)  =  20,  which  is  the  length  of  the  Y 
axis.  This  means  that  the  vehicle  will  traverse  the  entire  scene  in  most  cases  unless 
major  turns  are  incurred  from  the  guidance  control  logic.  Before  each  new  iteration 
is  started,  the  heading,  H  changes,  and  the  pitch  angle,  P  changes  are  updated  as 
(see  Section  13.5.41  for  0) 


H  =  H  +  <ph 

(4.6) 

P  =  P  +  (pv 

(4.7) 
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Figure  4.16:  High  Resolution  Optic  Flow  Field  Same  Position  as  in  Figi4.15l 


where  fa  denotes  the  control  signal  is  the  horizontal  channel  and  fa  denotes  the 
vertical  channel’s  control  signal.  The  new  heading  and  pitch  values  were  used  to 
update  the  vehicle  motion  vectors  in  the  following  way 


Tx 

=  Rcos(P)  cos(P) 

(4.8) 

T 

=  V  cos(P)  sm(H) 

(4.9) 

Tz 

=  V  sin(P) 

(4.10) 

Rin(  1) 

=  P 

(4.11) 

Rin(  3) 

(4.12) 
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where  V  is  the  magnitude  of  the  velocity.  This  ensures  that  the  total  velocity  remains 
constant.  Then  the  position  and  orientation  vectors  are  updated  as 


p  =  p  +T. 

1  v  1  v  '  -1-  in 

(4.13) 

Ry  Ry  Rin 

(4.14) 

After  these  are  updated,  the  whole  process  begins  again  and  repeats  until  the  vehicle 
completes  all  100  iterations  for  each  scenario. 


4-3  Results 

The  results  of  the  simulation  show  that  using  optic  flow  vector  averaging  and 
using  a  threshold  of  the  mean  plus  twice  the  standard  deviation  presents  the  best 
results  in  this  simulation  for  attaining  collision  avoidance.  The  method  of  curl  seg¬ 
mentation  and  using  averaging  with  a  threshold  of  the  twice  the  mean,  proved  to  be 
less  effective.  These  two  segmentation  methods  did  not  fail  completely  but  worked 
well  when  sufficient  ROIs  are  found.  The  results  in  Table  4.1  display  the  percentages 
of  avoidance  for  comparison. 


4-3.1  Curl  Segmentation.  The  curl  segmentation  method  does  not  perform 
as  well  as  expected.  This  is  primarily  due  to  the  the  fact  that  it  is  a  gradient-based 
operator.  This  means  that  the  curl  operator  finds  areas  of  gradient  changes,  like  edges, 
very  well.  The  operator  works  well  for  obstacles  that  are  narrow  but  not  for  obstacles 
that  are  considerably  wide.  Wide  obstacles  are  defined  as  those  that  have  more  than 
six  units  of  width.  For  these  obstacles,  the  curl  operator  identified  edges  left  and 
right  or  top  and  bottom  as  two  separate  obstacles  and  the  image  morphology  could 
not  combine  them.  This  is  because  of  large  pixel  distances  and  morphology  approach 
used,  which  connects  ROIs  that  are  within  two  pixels  of  each  other.  This  mistake 
guides  the  vehicle  directly  into  the  obstacles  and  a  collision  ensues.  The  increase  in 
resolution  from  low  resolution  to  high  only  exacerbated  this  problem.  This  is  because 
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Table  4.1:  Number  of  Collisions  by  Segmentation  Method  for  Each  Scene;  where 

each  number  along  the  top  row  designated  scene  number,  Tot.  represents  the  total 
number  of  collisions,  and  %  Avd.  is  the  percentage  of  avoidance  through  all  the  scenes 


Method 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

11 

12 

13 

14 

Tot. 

%  Avd. 

Curl 

Low  Res. 

0 

0 

3 

0 

8 

6 

0 

0 

0 

0 

0 

0 

8 

6 

31 

77.8 

Curl 

High  Res. 

1 

0 

6 

2 

6 

6 

2 

2 

2 

2 

2 

0 

4 

5 

40 

71.4 

Avg  2a 
Low  Res. 

0 

0 

1 

0 

9 

1 

0 

0 

0 

0 

0 

0 

6 

0 

17 

87.9 

Avg  20- 
High  Res. 

0 

0 

1 

0 

5 

2 

0 

0 

0 

0 

0 

0 

4 

1 

13 

90.7 

Avg  2  •  (i 
Low  Res. 

0 

0 

0 

0 

7 

9 

0 

0 

0 

0 

0 

0 

9 

1 

26 

81.4 

Avg  2  •  g 
High  Res. 

0 

0 

1 

0 

6 

8 

0 

0 

0 

0 

0 

0 

8 

0 

23 

83.6 

with  more  pixels  on  the  obstacle  it  takes  a  thinner  obstacle  to  morph  into  one  object. 
This  is  noted  by  the  increase  in  collisions  with  the  high  resolution  runs  using  this 
segmentation  method  -  see  Table  4.1.  Successful  navigation  through  a  scene  can  be 
seen  in  Fig.  14.171  but  note  that  the  obstacles  are  thin.  The  failure  can  be  seen  in 
Figs.  14.181  and  14.191  but  note  that  Fig.  14.181  is  for  low  resolution  and  Fig.  14.191  is 
for  the  high  resolution.  From  these  examples,  a  collision  takes  place  in  the  higher 
resolution  scenario  that  would  not  occur  in  the  lower  resolution  scenario. 

4-3.2  Averaging  Segmentation.  The  averaging  segmentation  method  works 
much  better  than  the  curl  method.  Averaging  of  the  optic  flow  vectors  is  better  at 
finding  significant  flow  vectors  caused  by  obstacles.  It  provided  ROIs  much  more 
representative  of  the  obstacles  for  the  vectors  that  were  above  the  given  threshold. 
The  thresholds  are  g+2a  and  2 // .  where  //  is  the  average  of  the  flow  vectors  in  the  scene 
and  cr  is  the  standard  deviation.  The  two  different  thresholds  perform  almost  equally 
well  when  bias  is  removed.  The  first  method  using  g  +  2cr  does  not  perform  as  well 
when  the  optic  flow  field  is  flooded  with  different  obstacles.  This  is  because  when  the 
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Figure  4.17:  navigation  success  with  curl  segmentation 
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Figure  4.18:  navigation  failure  using  curl  segmentation 

flow  field  is  full  of  varying  magnitudes  of  vectors,  the  distribution’s  standard  deviation 
grows  large.  This  means  that  using  two  standard  deviations  above  the  mean  gives 
no  obstacles  detected,  when  in  fact,  there  are  multiple  obstacles.  Even  though  the 
deviation  is  large,  using  2/j  for  a  threshold  that  increases  the  likelihood  that  the  closer 
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3-D  Scene 


Figure  4.19:  navigation  failure  using  curl  with  high  resolution 

obstacles  (i.e.  large  magnitude  vectors)  are  segmented.  In  this  case,  where  using  the 
standard  deviation  failed,  using  twice  the  mean  was  successful  -  see  Figs.  14.201 14.211 
and  14.221  The  second  method  entailing  the  use  of  2/j,  does  not  perform  well  when  the 
flow  field  is  mostly  populated  by  vectors  from  one  obstacle.  This  is  due  to  the  mean 
of  the  distribution  being  skewed  closer  to  the  magnitude  of  the  vectors  created  by  the 
obstacle.  Using  2 p  here  provides  no  obstacle  detection  because  2/t  falls  outside  the 
vector’s  distribution.  Using  /i  +  2a  at  this  point  would  result  in  obstacle  detection 
because  the  deviation  is  not  significant.  For  this  distinct,  case  the  second  method 
would  fail  when  the  first  one  would  be  successful  -  see  Figs.  14.231 14.241  and  14.251 
The  two  methods  perform  well  and  using  either  one  accomplishes  collision  obstacle 
avoidance  for  most  cases  -  see  Table  4.1.  The  method  using  p  +  2 a  has  a  slight  edge. 
Successful  trajectories  through  the  scenes  for  each  of  the  two  methods  can  be  seen 
in  Figs.  14.261  and  14.271  The  failure  of  fi  +  2a  thresholding  can  be  seen  in  Fig.  14.281 
but  the  success  of  2 //  thresholding  can  be  seen  for  the  similar  situation  in  Fig.  14.291 
The  failure  of  2/r  thresholding  can  be  seen  in  Fig.  14.301  but  the  success  of  p,  +  20- 
thresholding  can  be  seen  in  the  similar  situation  illustrated  in  Fig.  14.311  The  results 
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3-D  Scene 


Figure  4.20:  Scene  with  Trajectory  Depicting  the  Vehicles  Location  for  the  Follow¬ 
ing  Figs.  14.21 1  and  14.221 
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Figure  4.21:  Optic  Flow  Field  Displaying  Multiple  Obstacles  With  Centroid  (black 
asterisk)  Placed  on  the  Imminent  Obstacles  and  With  Looming  Detection  (red  aster¬ 
isks)  Using  2/i  for  Segmentation 


in  the  table  take  into  account  two  scenes  that  are  intended  to  show  the  limitations  of 
the  guidance  algorithm  for  obstacle  detection  using  optic  flow  measurement.  Scenes 
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Plot  of  the  Flow  Vector  Distributions 


Figure  4.22:  Plot  of  the  Distribution  of  the  Vector  Magnitudes  With  [i  +  2a  and  2/j 
Marked 

3-D  Scene 


Figure  4.23:  Scene  with  Trajectory  Depicting  the  Vehicles  Location  for  the  Follow¬ 
ing  Figs.  14.241  and  14.251 


5  and  6  contain  large  obstacles  that  would  cover  the  entire  camera’s  FOV’s  creating 
more  of  a  uniform  field  of  view,  and  the  algorithm  would  detect  no  obstacles.  If  these 


Optic  Flow  Field 


Figure  4.24:  Optic  Flow  Field  Displaying  One  Large  Obstacle  With  Centroid  (black 
asterisk)  Placed  on  the  Imminent  Obstacles  Using  /i  +  2a  for  Segmentation 


Plot  of  the  Flow  Vector  Distributions 


Figure  4.25:  Plot  of  the  Distribution  of  the  Vector  Magnitudes  With  /i  +  2 a  and  2/i 
Marked 

two  scenes  are  discarded,  the  probabilities  of  obstacle  avoidance  for  the  high  resolution 
methods  would  be  76.7%,  95.8%,  and  92.5%  for  the  curl,  n  +  2 a,  and  2/i  segmentation 
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3-D  Scene 
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Figure  4.26:  Successful  Navigation  Through  Urban  Scene  with  2/r.  The  elevation 
on  the  view  is  increased  to  get  total  sight  of  the  entire  trajectory.  This  gives  the 
buildings  a  strange  squatty  look 
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Figure  4.27:  Successful  Navigation  Through  Urban  Scene  With  /j,  +  2a.  Once  again 
the  elevation  is  increased  from  the  normal  view  to  see  the  entire  trajectory.  These 
building  also  look  squatty  as  a  result  of  the  look  angle. 

methods,  respectively.  This  is  valid  because,  although  this  is  a  limitation  for  the 
algorithm,  this  scenario  should  never  occur.  No  UAV  or  MAY  would  ever  initialize  a 
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3-D  Scene 


Figure  4.28:  Collision  Using  /i  +  2a 

3-D  Scene 


X  Position 

Figure  4.29:  Collision  Avoidance  Using  2/i 


mission  in  front  of  a  wall,  or  turn  90°  on  a  dime.  This  is  physically  impossible  due 
to  turn  radius  limitations.  Those  highly  unlikely  scenarios  would  be  the  only  way 
to  encounter  the  described  troublesome  scene.  The  percentages  seem  to  show  that 
/i  +  2cr  is  slightly  better  than  2/i,  but  the  first  one  performed  better  due  to  some 
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3-D  Scene 


Figure  4.30:  Collision  Using  2 p, 


Figure  4.31:  Collision  Avoidance  Using  /i  +  2a 
unexpected  effects  from  the  simulation  that  is  discussed  in  the  Section  14.3.31  These 
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artifacts  create  favorable  outcomes  for  the  trajectories  because  the  method  is  more 
sensitive  to  this  artifact. 

4-3.3  Unexpected  Incidents.  In  the  simulation,  there  were  a  few  things 
that  are  quite  unexpected  -  see  Figs  14 .321  and  14.331  After  analyzing  these  results  an 
interesting  insight  is  attained.  The  whole  scene  is  built  in  a  mesh  grid  that  is  a 
cube  of  20  x  20  x  20,  but  the  simulation  appears  as  if  it  is  flying  inside  a  box  of 
these  dimensions.  This  is  because  the  optic  flow  is  calculated  based  on  ranging,  as  in 
Section  13.5.21  and  anytime  that  the  vehicle  is  near  an  edge,  the  edge  is  perceived  as 
an  obstacle.  This  creates  the  false  sense  that  there  are  walls  around  the  entire  scene. 
This  can  create  strange  guidance  signals,  depending  on  where  it  finds  the  centroid 
on  these  walls.  Sometimes  they  arise  in  the  vertical  plane  and  sometimes  in  the 
horizontal  plane,  creating  trajectories  that  are  unwarranted.  These  anomalies  usually 
did  not  affect  the  outcomes  of  collisions  with  the  real  obstacles,  but  instead  seemed 
to  force  the  vehicle  to  change  altitude  or  direction  for  no  apparent  reason.  For  this 
reason,  these  trajectories  are  included  in  the  calculations  of  the  collisions,  but  note 
that  it  helps  some  scenes,  specifically  scenes  5  and  6,  by  making  changes  that  would 
not  normally  occur. 

4-4  Summary 

The  guidance  law  that  is  implemented  works  as  expected  and  many  successful 
trajectories  can  be  seen  in  Appendix  ??.  Collisions  can  be  avoided  with  this  scheme  in 
most  cases.  It  can  be  seen  though  that  there  are  limitations  to  the  guidance  law.  For 
instance,  the  scenes  with  the  walls  that  completely  covered  most  or  all  of  the  camera’s 
FOV,  created  a  collision  more  times  than  not  -  see  Table  4.1.  This  is  due  to  the  need 
for  contrast  in  the  scene.  There  must  be  features  that  can  be  accurately  tracked  over 
successive  frames  to  determine  looming  when  heading  directly  at  an  obstacle.  This 
means  obstacles  that  are  low  in  contrast,  especially  with  respect  to  the  background, 
will  be  problematic  for  this  algorithm. 
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3-D  Scene 


Figure  4.32:  Scene  With  a  Nonsense  Control  Implemented  based  on  Artifact  of  the 
Simulation 
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Figure  4.33:  Scene  With  a  Nonsense  Control  Implemented  based  on  Artifact  of  the 
Simulation 

An  important  thing  to  realize  is  that  image  processing  is  a  single  point  of  failure. 
The  method  chosen  to  segment  the  vectors  and  the  image  morphology  can  determine 
success  or  failure.  This  is  exemplified  by  the  failure  of  the  curl  segmentation  method, 
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where  areas  of  similar  magnitude  vectors  create  the  illusion  of  two  separate  obstacles 
when  only  one  exists.  Another  consideration  is  that  in  this  simulation  the  optic  flow 
created  is  noiseless,  which  means  that  with  real  imagery  the  optic  flow  might  not 
be  as  accurate.  This  makes  it  much  more  difficult  to  find  the  optic  flow  vectors 
associated  with  obstacles.  Although  there  may  have  been  isolated  failures  of  the 
different  methods  used  for  image  processing,  the  overall  success  rate  demonstrates 
that  LOS  rate  equalization  guidance  and  looming  detection  will  cause  a  UAV  to 
avoid  obstacles  autonomously. 
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V.  Conclusions 


5.1  Conclusion 

It  is  shown  that  intuitive  autonomous  guidance  schemes  for  obstacle  avoidance 
with  a  single  forward  facing  camera  based  on  nullifying  the  readily  measurable  LOS 
rate  difference  and  identifying  looming  based  on  optic  flow  sensing  will  yield  expected 
results.  In  fact,  the  flow  balancing  guidance  scheme  for  obstacle  avoidance  is  actually 
enhanced  with  the  inclusion  of  optical  sensor  limitations  such  as  FOV  and  range  lim¬ 
itations.  The  addition  of  looming  detection  strengthens  the  algorithm  to  account  for 
many  scenarios  of  obstacle  avoidance.  The  application  of  LOS  rate  balancing  guidance 
and  looming  detection  will  result  in  UAV  collision  avoidance.  The  symmetric  trajec¬ 
tories  are  stable,  although  the  linearized  dynamics  appear  to  indicate  otherwise.  The 
stability  can  be  seen  in  multiple  instances  of  trajectories  that  center  themselves  be¬ 
tween  two  obstacles  -  see  Appendix  ??.  Furthermore,  given  GPS/AHRS  information 
and  optic  flow  measurements,  range  to  obstacles  can  be  computed  and  the  obstacle 
avoidance  algorithm  will  be  enhanced,  especially  looming  detection.  Therefore,  guid¬ 
ance  based  on  the  measurement  of  optic  flow  and  LOS  rate  for  autonomous  obstacle 
avoidance  can  be  significantly  improved  if  the  MAV  is  equipped  with  a  GPS/AHRS 
(Attitude  and  Heading  Reference  System)  navigation  system.  One  can  then  use  the 
known  MAV  motion  and  the  optical  measurements  to  estimate  the  ranges  from  ob¬ 
stacles.  With  the  known  ranges,  a  guidance  law  can  be  developed  for  the  MAV  to 
fly  equidistant  between  obstacles  or  turn  away  from  looming  objects  in  front  of  the 
vehicle. 

This  research,  demonstrates  some  realistic  characteristics.  There  are  various 
other  issues  to  be  considered.  Optimally,  collision  avoidance  with  LOS  rate  equal¬ 
ization  should  occur  off  the  vertical  and  horizontal  axes  where  the  obstacles  generate 
optic  flow  in  locations  like  a  top- left  and  bottom-right  of  the  scene.  Another  is  that 
there  are  optic  flow  vectors  caused  by  the  vehicle’s  pitch  and  roll  that  need  to  be 
accounted  for  to  stabilize  the  optic  flow  and  pull  out  translational  trajectories  only. 
These  roll  motion  caused  optic  flow  vectors  can  create  an  illusion  of  motion,  especially 
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in  a  single  forward  facing  camera.  Next,  the  aircraft  has  a  six  degree  of  freedom  (6 
DOF)  of  motion  which  invariably  complicates  the  guidance  laws  required  for  colli¬ 
sion  avoidance.  This  is  simplified  in  this  work  to  a  two  channel  approach  with  two 
controls  independent  of  each  other.  A  real  system  would  have  to  incorporate  all  six 
controls  working  together.  Following  this,  real  systems  might  have  altitude  placards 
that  are  completely  ignored  in  this  work.  Finally,  one  of  the  most  important  issues 
is  the  real-time  calculation  of  optic  flow,  LOS  rate,  and  looming  for  actual  real-time 
operation. 

5.2  Future  Work 

The  future  of  collision  avoidance  for  UAVs  is  certain.  There  must  be  a  proved 
method  to  avoid  obstacles  and  it  must  be  accurate  and  reliable.  The  success  of 
missions  by  autonomous  UAVs  hinges  on  the  survivability  of  the  vehicle.  High  sur¬ 
vivability  rates  can  not  be  attained  without  competent  collision  avoidance.  More 
steps  must  be  taken  to  refine  this  work  for  the  intended  goal  of  an  errorless  collision 
avoidance  system.  Some  of  the  issues  that  need  to  be  addressed  are  the  following: 

•  Study  the  statistics  of  the  optic  flow  vectors  for  better  methods  of  finding  ob¬ 
stacles. 

•  Identify  the  best  optic  flow  algorithms  and  image  processing  for  segmentation. 

•  Implement  the  guidance  law  with  real  imagery. 

•  Add  more  complexity  to  the  simple  guidance,  e.g.,  non- frontal  looming  and  6 
DOF  motion. 

•  Add  range  calculation  from  video  for  time-to-collision  calculation  r. 

•  Finding  time  to  FOV  encapsulation  that  is  described  below 

As  described  in  section  14.3.21  the  statistics  of  mean  and  standard  deviation  are  used 
for  segmenting  the  the  optic  flow  vectors.  Even  though  these  worked  well,  further 
study  is  needed  using  pattern  recognition  methods  to  determine  a  precise  method  to 


97 


capture  the  distribution.  Points  of  interest  for  this  include  clustering,  expectation 
maximization,  or  a  Bayesian  classifier.  Using  various  optic  flow  routines  and  image 
processing  to  analyze  the  most  accurate  method  in  acquiring  the  best  flow  field  for 
segmenting  obstacle  vectors  from  background.  Some  items  that  should  be  explored  are 
the  number  of  flow  vectors  necessary,  the  method  for  calculation  i.e.,  Lucas-Kanade 
or  Horn-Schunck,  and  possibly  using  the  curl  operator  or  another  edge  detector  for 
looming  calculation  only.  There  should  be  a  study  to  determine  the  best  method  for 
capturing  actual  looming  in  real  imagery  as,  discussed  in  Appendix  ??.  Addition¬ 
ally,  the  guidance  control  law  can  be  implemented  with  real  imagery  and  a  UAV  for 
real  flight  testing  in  realistic  environments.  Once  real  imagery  is  used,  non- frontal 
looming  can  be  studied  for  veering  away  from  obstacles  that  are  encountered  in  real 
systems.  Non-frontal  looming  is  looming  that  occurs  in  the  right  or  left  and  not  the 
center  of  the  FOV.  This  would  allow  the  vehicle  to  saccade  away  from  the  object. 
Following  that,  installing  a  guidance  system  incorporating  six  degrees  of  freedom  for 
the  control  must  be  accomplished.  Lastly,  study  the  capability  to  back  out  ranging 
information  from  imagery  with  a  GPS/AHRS  system  on  board  the  air  vehicle  that 
can  give  critical  information  about  time  to  collision.  Time  to  collision  information 
can  enable  a  hierarchical  approach  to  the  avoidance  of  obstacles  in  the  scene.  Related 
to  this  is  the  time  to  FOV  encapsulation,  which  simply  incorporates  the  limitation 
of  this  algorithm.  This  limitation  is  the  inability  to  detect  obstacles  in  low  contrast 
images.  This  is  possible  if  the  obstacle  encapsulates  the  entire  FOV,  thus  a  collision 
becomes  unavoidable.  Therefore,  this  parameter  could  be  as  important  as  time-to- 
collision  because  with  no  contrast  a  UAV  will  carry  on  its  preplanned  course,  which 
might  mean  disaster. 

5. 3  Final  Thoughts 

The  capability  of  the  UAV  to  autonomously  accomplish  missions  without  risk¬ 
ing  human  lives  will  continue  to  drive  their  use.  As  real-time  image  processing  is 
becoming  increasingly  possible,  the  eventual  application  of  an  autonomous  UAV  col- 
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lision  avoidance  system  will  be  realized.  This  means  that  continued  research  on  this 
topic  is  necessary.  In  this  research  it  is  decisively  shown  in  simulation  that  collision 
avoidance  can  be  accomplished  with  a  basic  set  of  guidance  laws  incorporating  LOS 
rate  equalization  and  looming  detection. 
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Appendix  A.  Appendix  A:  Figures  Depicting  Successful  Trajectories 

A.l  Successful  Trajectories  With  the  Curl  Segementation 


3-D  Scene 


(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A.l:  successful  trajectories  through  scenel 
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(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A. 2:  successful  trajectories  through  scene2 


100 


3-D  Scene 


3-D  Scene 
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(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A. 3:  successful  trajectories  through  scene3 


(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A. 4:  successful  trajectories  through  scene4 


(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A. 5:  successful  trajectories  through  scene7 
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3-D  Scene 


3-D  Scene 


(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A. 6:  successful  trajectories  through  scene8 


(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A. 7:  successful  trajectories  through  scene9 


3-D  Scene  3-D  Scene 


(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A. 8:  successful  trajectories  through  scenelO 
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3-D  Scene 


3-D  Scene 


X  Position  X  Position 

(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A. 9:  successful  trajectories  through  scenell 


(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A.  10:  successful  trajectories  through  scenel2 


(a)  Trajectory  1 


(b)  Trajectory  2 


Figure  A.  11:  successful  trajectories  through  scenel3 
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3-D  Scene 


X  Position 

(a)  Trajectory  1 


Y  Position 

(b)  Trajectory  2 


Figure  A.  12: 


successful  trajectories  through  scenel4 


104 


A.  2  Successful  Trajectories  With  2/i  Segementation 


3-D  Scene  3-D  Scene 


(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A.  13:  successful  trajectories  through  scene  1 


3-D  Scene  3-D  Scene 


X  Position  X  Position 


(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A.  14:  successful  trajectories  through  scene2 
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3-D  Scene 
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(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A.  15:  successful  trajectories  through  scene3 
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(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A.  16:  successful  trajectories  through  scene4 
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(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A.  17:  successful  trajectories  through  scene7 
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(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A.  18:  successful  trajectories  through  scene8 
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(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A.  19:  successful  trajectories  through  scene9 


(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A. 20:  successful  trajectories  through  scenelO 
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3-D  Scene 


(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A. 21:  successful  trajectories  through  scenell 
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(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A. 22:  successful  trajectories  through  scene!2 
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(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A. 23:  successful  trajectories  through  scene!4 
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A. 3  Successful  Trajectories  With  the  g  +  2 a  Segementation 
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(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A. 24:  successful  trajectories  through  scene  1 
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(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A. 25:  successful  trajectories  through  scene2 
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(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A. 26:  successful  trajectories  through  scene3 
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(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A. 27:  successful  trajectories  through  scene4 


(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A. 28:  successful  trajectories  through  scene7 
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3-D  Scene 
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(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A. 29:  successful  trajectories  through  scene8 
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(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A. 30:  successful  trajectories  through  scene9 
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(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A. 31:  successful  trajectories  through  scenelO 
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(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A. 32:  successful  trajectories  through  scenell 


(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A. 33:  successful  trajectories  through  scene!2 


(a)  Trajectory  1  (b)  Trajectory  2 

Figure  A. 34:  successful  trajectories  through  scene!3 
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Figure  A. 35:  successful  trajectories  through  scene!4 
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Appendix  B.  Looming  With  Real  Optic  Flow 


B.  1  Introduction 

In  this  project,  a  sequence  of  video  and  the  use  of  image  processing  techniques, 
were  used  to  create  optic  flow  of  the  scene.  Optic  flow  was  accomplished  using  a 
correlation  based  motion  detection  algorithm.  This  was  accomplished  to  observe 
some  looming  in  the  motion  vectors.  Observing  these  motion  vectors  should  depict  if 
there  was  a  possibility  of  a  collision. 

To  achieve  this,  frames  of  video  will  be  separated  into  individual  images  and 
processed  frame  to  frame  for  change.  Comparisons  were  made  between  multiple  frame 
changes  to  determine  if  looming  is  occurring.  The  video  featured  a  square  or  rectangle 
object  that  a  camera  was  moving  toward  with  a  slight  amount  of  contrast  in  the 
background  (See  Fig.  ??). 


Frame  1  of  the  Image 


Figure  B.l:  Frame  1 
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Frame  2  of  the  Image 


Figure  B.2:  Frame  2 


B.2  Video  Sequence 

The  data  provided  for  this  project  was  a  video  sequence  collected  using  a  com¬ 
mercial  off  the  shelf,  Sony  high  definition  video  camcorder.  The  camcorder  collects 
video  at  a  standard  30 Hz  with  a  pixel  resolution  of  1440x1080.  Even  though  the 
aspect  ratio  is  16x9,  the  images  are  square  due  to  the  rectangular  shape  of  the  pixels 
in  each  image.  This  did  not  create  any  significant  problems  with  the  processing  of 
these  images. 

B.3  Pre-Processing  the  Video 

The  video  provided  had  to  be  converted  into  a  form  that  is  compatible  with 
Matlab.  The  raw  video  was  converted  to  AVI  files  using  freeware  called  Virtual  Dub. 
Once  they  were  in  this  format,  the  video  was  read  into  Matlab  with  the  AVI  READ 
command.  Due  to  the  size  of  the  images  and  the  length  of  the  video,  the  whole 
sequence  was  impossible  to  load  because  there  was  not  enough  memory  to  store  it. 
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To  alleviate  this  memory  issue,  every  tenth  frame  was  used  and  a  maximum  of  100 
frames  were  processed  at  any  one  time.  These  frames  were  saved  in  pairs  and  placed 
into  50  .Mat  hies  and  loaded  separately  for  quicker  computation.  This  also  allowed  for 
more  distinct  changes  in  the  image  pairs.  Next,  these  images  were  converted  into  gray 
scale  images,  contrast  enhanced  using  imadjust  command,  and  cropped  to  600x600 
resolution  (See  Fig.  ??).  This  gave  a  matrix  of  images  that  was  600  x  600  x  2.  This 
matrix  of  images  was  used  in  every  iteration  of  the  algorithm.  At  this  point  the  images 
were  ready  for  correlation  of  pixels  and  the  actual  optic  how  computation. 

Frame  1  of  the  Image  After  Enhancement 


I  1  •  V 

/ 


Figure  B.3:  Enhanced  Frame  1 

B.f  Computation  of  Optic  Flow 

The  Computation  of  the  optic  how  was  completed  in  the  Image  Corr  function. 
Each  image  was  broken  down  into  ’’super  pixels”  that  were  8x8  blocks  of  pixels.  This 
task  was  accomplished  with  the  Pixel  Cluster  function.  These  blocks  were  stripped 
from  frame  1  in  the  following  manner:  the  top  row  of  blocks  were  taken  hrst  from  left 
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Frame  2  of  the  Image  After  Enhancement 


Figure  B.4:  Enhanced  Frame  2 

to  right  and  then  work  down  each  8  block  row  until  the  bottom  edge  was  reached. 
The  index  to  the  original  image  for  each  block  was  kept  and  set  into  a  new  variable 
which  was  output  from  the  function  as  well  as  the  new  pixel  blocks. 

The  next  step  was  to  throw  away  any  super  pixels  that  do  not  have  any  in¬ 
tensity  information.  The  lack  of  contrast  makes  it  difficult  to  correlate  these  pixels 
to  any  other  pixels  because  of  the  vast  amount  of  benign  background.  To  finish  this 
assignment,  each  super  pixel  was  sent  into  the  Discard  function.  In  this  function,  the 
mean  intensity  for  each  block  was  found  and  subtracted  from  every  pixel  and  then  the 
pixels  were  summed  and  normalized  to  give  a  threshold  value.  If  the  threshold  was 
above  0.03,  then  the  block  was  kept  for  future  processing.  The  idea  is  to  minimize 
computations  by  discarding  regions  that  are  low  in  contrast.  This  minimized  the 
number  of  blocks  kept  which  also  helped  reduce  processing  time.  The  indexes  were 
all  maintained  through  this  process. 

A  minimized  number  of  blocks  with  intensity  information  now  exist  and  the 
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cross  correlation  can  take  place  accurately.  The  super  pixel  was  cross  correlated 
against  frame  2  of  the  pair  of  frames  to  determine  where  the  pixels  are  in  the  next 
frame.  This  was  achieved  using  the  Normxcorr2  function  in  Matlab.  This  function 
slides  the  super  pixel  block  over  the  image  and  calculates  the  cross  correlation  at  each 
point.  The  location  where  the  maximum  correlation  occurs  represents  the  best  new 
location  for  the  pixel  block  in  the  next  frame.  This  function  was  done  incrementally 
until  all  of  the  blocks  were  correlated. 

B.  5  Visualization 

The  final  step  was  to  visualize  the  optic  flow  by  creating  a  flow  field  using  the 
Qplot  function.  This  function  accepted  the  indexes  of  pixels  for  the  first  frame  and 
where  they  are  predicted  to  be  in  the  next  frame  then  calculated  the  difference  in  the 
x  and  y  distances.  In  order  to  emphasize  the  looming  effect  from  the  flow,  the  delta 
values  for  each  direction  were  averaged  and  this  average  was  subtracted  from  each 
delta  value.  This  minimized  the  translational  flow  in  the  flow  field  thus  depicting  more 
of  the  looming  effect.  Next  the  delta’s  are  multiplied  by  a  scale  factor  to  amplify  the 
flow  field  thus  allowing  more  discrimination.  Finally,  this  function  took  the  original 
pixel  coordinates  and  the  new  delta’s  and  used  the  Quiver  command  in  Matlab  to 
create  the  flow  field  (see  Fig.  ??). 

B.  6  Conclusions 

Several  observations  can  be  drawn  from  the  data  generated  by  the  process  de¬ 
scribed  above.  First,  this  correlation  based  method  was  highly  computationally  in¬ 
tensive.  Many  mitigating  factors  were  used  to  speed  up  the  processing.  Next,  there 
was  a  lot  of  blocks  of  pixels  that  contained  very  little  information,  and  a  threshold 
value  was  used  to  eliminate  those  blocks.  Also,  skipping  up  to  10  frames  gave  a  better 
optic  flow  output  as  larger  changes  in  the  imagery  were  present  over  the  longer  period 
of  sampling.  This  was  most  likely  dependent  on  the  data  that  is  being  analyzed. 
Lastly,  the  translational  flow  made  it  hard  to  discern  any  looming.  This  was  handled 
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Optic  Flowfield  with  Mode  Subtracted 


Figure  B.5:  Looming  FlowField 

as  previously  mentioned  with  the  mean  values  of  the  flow  vectors  being  subtracted  to 
null  the  translation  flow  as  much  as  possible. 
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